Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250110 20:56:48.891899 4783 test_util.cc:274] Using random seed: 1431773051
2025-01-10T20:56:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:56:48Z Disabled control of system clock
I20250110 20:56:48.955094 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.254:34617
--webserver_interface=127.4.171.254
--webserver_port=0
--builtin_ntp_servers=127.4.171.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250110 20:56:49.234911 4792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:56:49.235471 4792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:56:49.235913 4792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:56:49.265573 4792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:56:49.265856 4792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:56:49.266113 4792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:56:49.266343 4792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:56:49.299350 4792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.254:34617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.171.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:56:49.300597 4792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:56:49.302112 4792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:56:49.315882 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:56:49.316094 4798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:49.317600 4792 server_base.cc:1034] running on GCE node
W20250110 20:56:49.316893 4801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:50.392472 4792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:56:50.394940 4792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:56:50.396283 4792 hybrid_clock.cc:648] HybridClock initialized: now 1736542610396262 us; error 37 us; skew 500 ppm
I20250110 20:56:50.397071 4792 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:56:50.403306 4792 webserver.cc:458] Webserver started at http://127.4.171.254:41505/ using document root <none> and password file <none>
I20250110 20:56:50.404189 4792 fs_manager.cc:362] Metadata directory not provided
I20250110 20:56:50.404394 4792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:56:50.404814 4792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:56:50.409547 4792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data/instance:
uuid: "63aa77ee6ec84a6e82132b2de9e77486"
format_stamp: "Formatted at 2025-01-10 20:56:50 on dist-test-slave-98kh"
I20250110 20:56:50.410584 4792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/wal/instance:
uuid: "63aa77ee6ec84a6e82132b2de9e77486"
format_stamp: "Formatted at 2025-01-10 20:56:50 on dist-test-slave-98kh"
I20250110 20:56:50.417615 4792 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20250110 20:56:50.422623 4808 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:50.423578 4792 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250110 20:56:50.423874 4792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/wal
uuid: "63aa77ee6ec84a6e82132b2de9e77486"
format_stamp: "Formatted at 2025-01-10 20:56:50 on dist-test-slave-98kh"
I20250110 20:56:50.424192 4792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:56:50.469568 4792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:56:50.470954 4792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:56:50.471386 4792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:56:50.543422 4792 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.254:34617
I20250110 20:56:50.543483 4859 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.254:34617 every 8 connection(s)
I20250110 20:56:50.546020 4792 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/data/info.pb
I20250110 20:56:50.551635 4860 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:56:50.554234 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 4792
I20250110 20:56:50.554625 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-0/wal/instance
I20250110 20:56:50.558121 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.253:34381
--webserver_interface=127.4.171.253
--webserver_port=0
--builtin_ntp_servers=127.4.171.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 20:56:50.563126 4860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:56:50.588879 4860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } has no permanent_uuid. Determining permanent_uuid...
W20250110 20:56:50.591077 4812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
W20250110 20:56:50.595043 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:34381: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
I20250110 20:56:50.644594 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } attempt: 1
W20250110 20:56:50.648259 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:34381: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
I20250110 20:56:50.716722 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } attempt: 2
W20250110 20:56:50.720382 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:34381: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
I20250110 20:56:50.812372 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } attempt: 3
W20250110 20:56:50.816459 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:34381: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
W20250110 20:56:50.856067 4862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:56:50.856585 4862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:56:50.856966 4862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:56:50.890092 4862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:56:50.890354 4862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:56:50.890552 4862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:56:50.890743 4862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:56:50.924521 4862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.253:34381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.171.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:56:50.925830 4862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:56:50.927301 4862 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:56:50.943257 4872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:50.959858 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } attempt: 4
W20250110 20:56:50.963817 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:34381: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
I20250110 20:56:51.263185 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } attempt: 5
W20250110 20:56:51.267455 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:34381: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
I20250110 20:56:51.814832 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } attempt: 6
W20250110 20:56:51.821092 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:34381: Network error: Client connection negotiation failed: client connection to 127.4.171.253:34381: connect: Connection refused (error 111)
W20250110 20:56:50.943763 4871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:56:50.944828 4874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:50.946529 4862 server_base.cc:1034] running on GCE node
I20250110 20:56:52.037248 4862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:56:52.039741 4862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:56:52.041070 4862 hybrid_clock.cc:648] HybridClock initialized: now 1736542612041028 us; error 53 us; skew 500 ppm
I20250110 20:56:52.041872 4862 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:56:52.052768 4862 webserver.cc:458] Webserver started at http://127.4.171.253:40251/ using document root <none> and password file <none>
I20250110 20:56:52.053686 4862 fs_manager.cc:362] Metadata directory not provided
I20250110 20:56:52.053894 4862 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:56:52.054306 4862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:56:52.059027 4862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data/instance:
uuid: "b84772eba3f0460faf8523f76eb57e53"
format_stamp: "Formatted at 2025-01-10 20:56:52 on dist-test-slave-98kh"
I20250110 20:56:52.060052 4862 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/wal/instance:
uuid: "b84772eba3f0460faf8523f76eb57e53"
format_stamp: "Formatted at 2025-01-10 20:56:52 on dist-test-slave-98kh"
I20250110 20:56:52.067145 4862 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250110 20:56:52.072268 4883 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:52.073197 4862 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:56:52.073530 4862 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/wal
uuid: "b84772eba3f0460faf8523f76eb57e53"
format_stamp: "Formatted at 2025-01-10 20:56:52 on dist-test-slave-98kh"
I20250110 20:56:52.073844 4862 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:56:52.123528 4862 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:56:52.124899 4862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:56:52.125312 4862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:56:52.199276 4862 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.253:34381
I20250110 20:56:52.199349 4934 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.253:34381 every 8 connection(s)
I20250110 20:56:52.201825 4862 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
I20250110 20:56:52.207389 4935 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:56:52.211014 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 4862
I20250110 20:56:52.211476 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-1/wal/instance
I20250110 20:56:52.214612 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.252:37233
--webserver_interface=127.4.171.252
--webserver_port=0
--builtin_ntp_servers=127.4.171.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 20:56:52.221948 4935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:56:52.242805 4935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:56:52.256035 4935 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } has no permanent_uuid. Determining permanent_uuid...
W20250110 20:56:52.258097 4887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
W20250110 20:56:52.262099 4935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:52.311506 4935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 1
W20250110 20:56:52.315573 4935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:52.383930 4935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 2
W20250110 20:56:52.387683 4935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:52.479044 4935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 3
W20250110 20:56:52.482864 4935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
W20250110 20:56:52.511566 4937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:56:52.512112 4937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:56:52.512550 4937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:56:52.543495 4937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:56:52.543783 4937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:56:52.544020 4937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:56:52.544245 4937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:56:52.578574 4937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.252:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.171.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:56:52.579860 4937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:56:52.581357 4937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:56:52.596483 4947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:52.626206 4935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 4
W20250110 20:56:52.629974 4935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:52.881623 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } attempt: 7
I20250110 20:56:52.893507 4860 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } has no permanent_uuid. Determining permanent_uuid...
W20250110 20:56:52.898839 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:52.929513 4935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 5
W20250110 20:56:52.933228 4935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:52.957163 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 1
W20250110 20:56:52.961004 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:53.042379 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 2
W20250110 20:56:53.046160 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:53.131491 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 3
W20250110 20:56:53.136155 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:53.276647 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 4
W20250110 20:56:53.280588 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:53.480719 4935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 6
W20250110 20:56:53.499060 4935 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:56:53.564059 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 5
W20250110 20:56:53.568428 4860 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:37233: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
W20250110 20:56:52.596956 4948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:56:52.597625 4950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:52.598878 4937 server_base.cc:1034] running on GCE node
I20250110 20:56:53.686568 4937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:56:53.689926 4937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:56:53.691349 4937 hybrid_clock.cc:648] HybridClock initialized: now 1736542613691285 us; error 78 us; skew 500 ppm
I20250110 20:56:53.692093 4937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:56:53.698019 4937 webserver.cc:458] Webserver started at http://127.4.171.252:45215/ using document root <none> and password file <none>
I20250110 20:56:53.698858 4937 fs_manager.cc:362] Metadata directory not provided
I20250110 20:56:53.699033 4937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:56:53.699424 4937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:56:53.703998 4937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/instance:
uuid: "14d1e2d5c72f498f915c367e8750361f"
format_stamp: "Formatted at 2025-01-10 20:56:53 on dist-test-slave-98kh"
I20250110 20:56:53.705008 4937 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal/instance:
uuid: "14d1e2d5c72f498f915c367e8750361f"
format_stamp: "Formatted at 2025-01-10 20:56:53 on dist-test-slave-98kh"
I20250110 20:56:53.711630 4937 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250110 20:56:53.717036 4962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:53.718004 4937 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250110 20:56:53.718289 4937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
uuid: "14d1e2d5c72f498f915c367e8750361f"
format_stamp: "Formatted at 2025-01-10 20:56:53 on dist-test-slave-98kh"
I20250110 20:56:53.718600 4937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:56:53.778334 4937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:56:53.779707 4937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:56:53.780113 4937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:56:53.851548 4937 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.252:37233
I20250110 20:56:53.851621 5013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.252:37233 every 8 connection(s)
I20250110 20:56:53.854122 4937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
I20250110 20:56:53.854627 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 4937
I20250110 20:56:53.855055 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal/instance
I20250110 20:56:53.860188 5014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:56:53.878082 5014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:56:53.901058 5014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:56:53.911314 5014 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:56:53.928184 5014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Bootstrap starting.
I20250110 20:56:53.934612 5014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Neither blocks nor log segments found. Creating new log.
I20250110 20:56:53.936167 5014 log.cc:826] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Log is configured to *not* fsync() on all Append() calls
I20250110 20:56:53.940349 5014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: No bootstrap required, opened a new log
I20250110 20:56:53.961562 5014 raft_consensus.cc:357] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:53.962142 5014 raft_consensus.cc:383] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:56:53.962366 5014 raft_consensus.cc:738] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d1e2d5c72f498f915c367e8750361f, State: Initialized, Role: FOLLOWER
I20250110 20:56:53.963055 5014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:53.965978 5025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:53.966539 5025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:56:53.966842 5014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: configured and running, proceeding with master startup.
W20250110 20:56:53.993870 5036 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:56:53.994294 5036 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:56:53.986362 4937 master_runner.cc:186] Error getting master registration for 127.4.171.254:34617: OK, instance_id { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" instance_seqno: 1736542610523399 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250110 20:56:54.012355 4937 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250110 20:56:54.035193 4968 tablet.cc:2367] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:56:54.120800 4860 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 6
I20250110 20:56:54.136827 4860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: Bootstrap starting.
I20250110 20:56:54.143095 4860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: Neither blocks nor log segments found. Creating new log.
I20250110 20:56:54.144667 4860 log.cc:826] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: Log is configured to *not* fsync() on all Append() calls
I20250110 20:56:54.148944 4860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: No bootstrap required, opened a new log
I20250110 20:56:54.180184 4860 raft_consensus.cc:357] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:54.181052 4860 raft_consensus.cc:383] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:56:54.181365 4860 raft_consensus.cc:738] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63aa77ee6ec84a6e82132b2de9e77486, State: Initialized, Role: FOLLOWER
I20250110 20:56:54.182274 4860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:54.185264 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.185896 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:56:54.187304 4860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: configured and running, proceeding with master startup.
W20250110 20:56:54.229605 4814 tablet.cc:2367] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:56:54.239498 5051 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:56:54.239926 5051 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:56:54.220710 4792 master_runner.cc:186] Error getting master registration for 127.4.171.253:34381: OK, instance_id { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" instance_seqno: 1736542612178955 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250110 20:56:54.254544 4792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250110 20:56:54.270546 5025 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:56:54.271073 5025 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:54.273453 5025 leader_election.cc:290] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381)
I20250110 20:56:54.274785 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:56:54.275501 4835 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14d1e2d5c72f498f915c367e8750361f in term 0.
I20250110 20:56:54.277122 4966 leader_election.cc:304] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486; no voters:
I20250110 20:56:54.278061 5025 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:56:54.278393 5025 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:56:54.278687 5025 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:56:54.285164 5025 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:54.288924 5025 leader_election.cc:290] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 1 election: Requested vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381)
I20250110 20:56:54.290577 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486"
I20250110 20:56:54.291118 4835 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:56:54.297456 4835 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14d1e2d5c72f498f915c367e8750361f in term 1.
I20250110 20:56:54.298481 4966 leader_election.cc:304] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486; no voters:
I20250110 20:56:54.299444 5025 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:56:54.309835 5025 raft_consensus.cc:695] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 1 LEADER]: Becoming Leader. State: Replica: 14d1e2d5c72f498f915c367e8750361f, State: Running, Role: LEADER
I20250110 20:56:54.310783 5025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:54.318373 5053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d1e2d5c72f498f915c367e8750361f. Latest consensus state: current_term: 1 leader_uuid: "14d1e2d5c72f498f915c367e8750361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.319155 5053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: This master's current role is: LEADER
I20250110 20:56:54.328420 5055 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:56:54.336238 5055 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:56:54.358837 4835 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Refusing update from remote peer 14d1e2d5c72f498f915c367e8750361f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:56:54.360502 5053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [LEADER]: Connected to new peer: Peer: permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:56:54.392719 5056 mvcc.cc:204] Tried to move back new op lower bound from 7112878548386652160 to 7112878548227158016. Current Snapshot: MvccSnapshot[applied={T|T < 7112878548386652160}]
I20250110 20:56:54.396625 5040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d1e2d5c72f498f915c367e8750361f. Latest consensus state: current_term: 1 leader_uuid: "14d1e2d5c72f498f915c367e8750361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.399226 5025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14d1e2d5c72f498f915c367e8750361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.400010 5025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: This master's current role is: LEADER
I20250110 20:56:54.397351 5040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:56:54.403477 5057 mvcc.cc:204] Tried to move back new op lower bound from 7112878548386652160 to 7112878548227158016. Current Snapshot: MvccSnapshot[applied={T|T < 7112878548386652160}]
I20250110 20:56:54.406832 5055 catalog_manager.cc:1348] Generated new cluster ID: 9189b87cd4594c99bd9b29530bf32da1
I20250110 20:56:54.407097 5055 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:56:54.412560 5060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14d1e2d5c72f498f915c367e8750361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.413179 5060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:56:54.433506 5055 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:56:54.435446 5055 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:56:54.461002 5055 catalog_manager.cc:5899] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Generated new TSK 0
I20250110 20:56:54.462064 5055 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:56:54.492805 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.171.193:0
--local_ip_for_outbound_sockets=127.4.171.193
--webserver_interface=127.4.171.193
--webserver_port=0
--tserver_master_addrs=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--builtin_ntp_servers=127.4.171.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 20:56:54.559672 4935 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } attempt: 7
I20250110 20:56:54.606913 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Bootstrap starting.
I20250110 20:56:54.614205 4935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Neither blocks nor log segments found. Creating new log.
I20250110 20:56:54.628036 4935 log.cc:826] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Log is configured to *not* fsync() on all Append() calls
I20250110 20:56:54.652031 4935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: No bootstrap required, opened a new log
I20250110 20:56:54.704360 4935 raft_consensus.cc:357] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:54.714460 4935 raft_consensus.cc:383] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:56:54.714854 4935 raft_consensus.cc:738] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84772eba3f0460faf8523f76eb57e53, State: Initialized, Role: FOLLOWER
I20250110 20:56:54.715857 4935 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:54.722328 5065 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.723044 5065 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:56:54.726198 4935 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:56:54.726680 4908 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:56:54.729892 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:56:54.730115 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53"
I20250110 20:56:54.737720 4908 raft_consensus.cc:1270] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Refusing update from remote peer 14d1e2d5c72f498f915c367e8750361f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:56:54.739243 5053 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:56:54.795019 5065 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14d1e2d5c72f498f915c367e8750361f. Latest consensus state: current_term: 1 leader_uuid: "14d1e2d5c72f498f915c367e8750361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.796571 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "14d1e2d5c72f498f915c367e8750361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:54.797757 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:56:54.797031 5065 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:56:54.858352 5082 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250110 20:56:54.952800 5062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:56:54.953225 5062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:56:54.953804 5062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:56:55.010298 5062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:56:55.011844 5062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.171.193
I20250110 20:56:55.073096 5062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.171.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.171.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.171.193
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:56:55.074815 5062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:56:55.076797 5062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:56:55.098201 5088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:55.159224 5094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "14d1e2d5c72f498f915c367e8750361f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:56:55.159986 5094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: This master's current role is: LEADER
I20250110 20:56:55.246523 5051 catalog_manager.cc:1260] Loaded cluster ID: 9189b87cd4594c99bd9b29530bf32da1
I20250110 20:56:55.246942 5051 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: loading cluster ID for follower catalog manager: success
I20250110 20:56:55.265548 5051 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: acquiring CA information for follower catalog manager: success
I20250110 20:56:55.272459 5051 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 20:56:55.466747 5097 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:55.467480 5097 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:55.475452 5097 leader_election.cc:290] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:55.477097 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:56:55.478545 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:55.480713 4812 leader_election.cc:304] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486; no voters: 14d1e2d5c72f498f915c367e8750361f, b84772eba3f0460faf8523f76eb57e53
I20250110 20:56:55.481534 5097 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:55.874382 5082 catalog_manager.cc:1260] Loaded cluster ID: 9189b87cd4594c99bd9b29530bf32da1
I20250110 20:56:55.880175 5082 catalog_manager.cc:1553] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: loading cluster ID for follower catalog manager: success
I20250110 20:56:55.892277 5082 catalog_manager.cc:1575] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: acquiring CA information for follower catalog manager: success
I20250110 20:56:55.898600 5082 catalog_manager.cc:1603] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 20:56:56.041167 5099 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:56.042991 5099 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:56.045840 5099 leader_election.cc:290] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:56.047518 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:56:56.048774 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:56.050823 4812 leader_election.cc:304] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486; no voters: 14d1e2d5c72f498f915c367e8750361f, b84772eba3f0460faf8523f76eb57e53
I20250110 20:56:56.051534 5099 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:56.558749 5100 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:56.559396 5100 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:56.566356 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:56:56.566859 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:56.568229 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b84772eba3f0460faf8523f76eb57e53; no voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486
I20250110 20:56:56.579614 5100 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:56.580179 5100 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:56:56.493698 5087 debug-util.cc:398] Leaking SignalData structure 0x7b080001c400 after lost signal to thread 5062
W20250110 20:56:56.937834 5087 kernel_stack_watchdog.cc:198] Thread 5062 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 393ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250110 20:56:55.099098 5089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:56:56.957075 5062 thread.cc:635] OpenStack (cloud detector) Time spent creating pthread: real 1.859s user 0.003s sys 0.000s
W20250110 20:56:56.957445 5062 thread.cc:601] OpenStack (cloud detector) Time spent starting thread: real 1.859s user 0.003s sys 0.000s
W20250110 20:56:56.958524 5090 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1857 milliseconds
W20250110 20:56:56.959340 5091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:56.959780 5062 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250110 20:56:56.961098 5062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:56:56.969856 5062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:56:56.977523 5062 hybrid_clock.cc:648] HybridClock initialized: now 1736542616977467 us; error 48 us; skew 500 ppm
I20250110 20:56:56.978615 5062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:56:57.010090 5062 webserver.cc:458] Webserver started at http://127.4.171.193:39111/ using document root <none> and password file <none>
I20250110 20:56:57.011340 5062 fs_manager.cc:362] Metadata directory not provided
I20250110 20:56:57.011619 5062 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:56:57.012157 5062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:56:57.018973 5062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data/instance:
uuid: "ac9003be04f340f3886b5625cc928eb0"
format_stamp: "Formatted at 2025-01-10 20:56:57 on dist-test-slave-98kh"
I20250110 20:56:57.019227 5105 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:57.020385 5062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/instance:
uuid: "ac9003be04f340f3886b5625cc928eb0"
format_stamp: "Formatted at 2025-01-10 20:56:57 on dist-test-slave-98kh"
I20250110 20:56:57.019997 5105 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:57.022825 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:56:57.028717 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:57.030190 4812 leader_election.cc:304] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486; no voters: 14d1e2d5c72f498f915c367e8750361f, b84772eba3f0460faf8523f76eb57e53
I20250110 20:56:57.034682 5062 fs_manager.cc:696] Time spent creating directory manager: real 0.014s user 0.007s sys 0.009s
I20250110 20:56:57.035629 5105 leader_election.cc:290] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:57.036562 5105 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:57.047767 5108 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:57.048906 5062 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.001s sys 0.007s
I20250110 20:56:57.049305 5062 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal
uuid: "ac9003be04f340f3886b5625cc928eb0"
format_stamp: "Formatted at 2025-01-10 20:56:57 on dist-test-slave-98kh"
I20250110 20:56:57.049861 5062 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:56:57.156216 5062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:56:57.158200 5062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:56:57.158835 5062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:56:57.167798 5062 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:56:57.191392 5062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:56:57.191787 5062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250110 20:56:57.192162 5062 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:56:57.192420 5062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:57.455169 5190 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:57.455904 5190 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:57.457950 5190 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:57.458950 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:56:57.460691 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:57.461923 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b84772eba3f0460faf8523f76eb57e53; no voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486
I20250110 20:56:57.462605 5190 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:57.503386 5062 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.193:36063
I20250110 20:56:57.503553 5221 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.193:36063 every 8 connection(s)
I20250110 20:56:57.505838 5062 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/data/info.pb
I20250110 20:56:57.512862 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 5062
I20250110 20:56:57.513362 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/instance
I20250110 20:56:57.526546 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.171.194:0
--local_ip_for_outbound_sockets=127.4.171.194
--webserver_interface=127.4.171.194
--webserver_port=0
--tserver_master_addrs=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--builtin_ntp_servers=127.4.171.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 20:56:57.594521 5224 heartbeater.cc:346] Connected to a master server at 127.4.171.254:34617
I20250110 20:56:57.595005 5224 heartbeater.cc:463] Registering TS with master...
I20250110 20:56:57.596158 5224 heartbeater.cc:510] Master 127.4.171.254:34617 requested a full tablet report, sending...
I20250110 20:56:57.597340 5222 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:56:57.597697 5222 heartbeater.cc:463] Registering TS with master...
I20250110 20:56:57.598485 5222 heartbeater.cc:510] Master 127.4.171.252:37233 requested a full tablet report, sending...
I20250110 20:56:57.599733 5223 heartbeater.cc:346] Connected to a master server at 127.4.171.253:34381
I20250110 20:56:57.600047 5223 heartbeater.cc:463] Registering TS with master...
I20250110 20:56:57.600821 5223 heartbeater.cc:510] Master 127.4.171.253:34381 requested a full tablet report, sending...
I20250110 20:56:57.604190 4825 ts_manager.cc:194] Registered new tserver with Master: ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:56:57.604614 4900 ts_manager.cc:194] Registered new tserver with Master: ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:56:57.605973 4979 ts_manager.cc:194] Registered new tserver with Master: ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:56:57.607765 4979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.171.193:48745
W20250110 20:56:57.862430 5229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:56:57.863044 5229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:56:57.863941 5229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:56:57.904281 5229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:56:57.905283 5229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.171.194
I20250110 20:56:57.939150 5229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.171.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.171.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.171.194
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:56:57.940438 5229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:56:57.941954 5229 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:56:57.960855 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:57.992164 5244 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:57.992556 5244 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:57.994311 5244 leader_election.cc:290] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:57.995195 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:56:57.995613 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:57.997231 4812 leader_election.cc:304] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486; no voters: 14d1e2d5c72f498f915c367e8750361f, b84772eba3f0460faf8523f76eb57e53
I20250110 20:56:57.998179 5244 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:58.022327 5245 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:58.022712 5245 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:58.024152 5245 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:58.025216 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:58.025254 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:56:58.026757 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b84772eba3f0460faf8523f76eb57e53; no voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486
I20250110 20:56:58.027772 5245 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:58.111047 5222 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
I20250110 20:56:58.515113 5246 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:58.515599 5246 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:58.518187 5246 leader_election.cc:290] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:58.518329 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:56:58.519910 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:58.522338 4812 leader_election.cc:304] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486; no voters: 14d1e2d5c72f498f915c367e8750361f, b84772eba3f0460faf8523f76eb57e53
I20250110 20:56:58.523499 5246 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:58.624177 5247 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:58.625113 5247 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:58.628214 5247 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:58.629146 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:56:58.629631 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:58.632905 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b84772eba3f0460faf8523f76eb57e53; no voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486
I20250110 20:56:58.633765 5247 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:56:57.961002 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:56:57.961575 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:56:59.069489 5240 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1109 milliseconds
I20250110 20:56:59.069597 5229 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250110 20:56:59.073448 5229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:56:59.075838 5229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:56:59.077255 5229 hybrid_clock.cc:648] HybridClock initialized: now 1736542619077223 us; error 39 us; skew 500 ppm
I20250110 20:56:59.078081 5229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:56:59.084499 5229 webserver.cc:458] Webserver started at http://127.4.171.194:44151/ using document root <none> and password file <none>
I20250110 20:56:59.085428 5229 fs_manager.cc:362] Metadata directory not provided
I20250110 20:56:59.085646 5229 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:56:59.086061 5229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:56:59.090867 5229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data/instance:
uuid: "46005a5920cc4a3697a5c61b1c271384"
format_stamp: "Formatted at 2025-01-10 20:56:59 on dist-test-slave-98kh"
I20250110 20:56:59.091907 5229 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/instance:
uuid: "46005a5920cc4a3697a5c61b1c271384"
format_stamp: "Formatted at 2025-01-10 20:56:59 on dist-test-slave-98kh"
I20250110 20:56:59.099005 5229 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250110 20:56:59.103814 5253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:59.104766 5229 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250110 20:56:59.105058 5229 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal
uuid: "46005a5920cc4a3697a5c61b1c271384"
format_stamp: "Formatted at 2025-01-10 20:56:59 on dist-test-slave-98kh"
I20250110 20:56:59.105352 5229 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:56:59.159511 5229 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:56:59.160854 5229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:56:59.161226 5229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:56:59.163913 5229 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:56:59.168179 5229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:56:59.168399 5229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:59.168625 5229 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:56:59.168761 5229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:56:59.202602 5280 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:56:59.203143 5280 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:56:59.205031 5280 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:56:59.206562 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:56:59.206629 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:56:59.208243 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b84772eba3f0460faf8523f76eb57e53; no voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486
I20250110 20:56:59.208845 5280 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:56:59.318351 5229 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.194:40295
I20250110 20:56:59.318441 5367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.194:40295 every 8 connection(s)
I20250110 20:56:59.320731 5229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/data/info.pb
I20250110 20:56:59.330338 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 5229
I20250110 20:56:59.331027 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/instance
I20250110 20:56:59.353147 5368 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:56:59.353612 5368 heartbeater.cc:463] Registering TS with master...
I20250110 20:56:59.354902 5368 heartbeater.cc:510] Master 127.4.171.252:37233 requested a full tablet report, sending...
I20250110 20:56:59.355278 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.171.195:0
--local_ip_for_outbound_sockets=127.4.171.195
--webserver_interface=127.4.171.195
--webserver_port=0
--tserver_master_addrs=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--builtin_ntp_servers=127.4.171.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 20:56:59.358625 4979 ts_manager.cc:194] Registered new tserver with Master: 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:56:59.360201 4979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.171.194:52633
I20250110 20:56:59.362319 5369 heartbeater.cc:346] Connected to a master server at 127.4.171.253:34381
I20250110 20:56:59.362638 5369 heartbeater.cc:463] Registering TS with master...
I20250110 20:56:59.363209 5369 heartbeater.cc:510] Master 127.4.171.253:34381 requested a full tablet report, sending...
I20250110 20:56:59.365653 4900 ts_manager.cc:194] Registered new tserver with Master: 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:56:59.369019 5370 heartbeater.cc:346] Connected to a master server at 127.4.171.254:34617
I20250110 20:56:59.369288 5370 heartbeater.cc:463] Registering TS with master...
I20250110 20:56:59.369800 5370 heartbeater.cc:510] Master 127.4.171.254:34617 requested a full tablet report, sending...
I20250110 20:56:59.372164 4825 ts_manager.cc:194] Registered new tserver with Master: 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:56:59.654641 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:56:59.655090 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:56:59.655654 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:56:59.685351 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:56:59.686260 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.171.195
I20250110 20:56:59.720180 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.171.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.171.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.171.195
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:56:59.721376 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:56:59.722857 5378 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:56:59.740000 5384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:59.864035 5368 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
W20250110 20:56:59.741001 5387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:56:59.741762 5378 server_base.cc:1034] running on GCE node
W20250110 20:56:59.740826 5385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:00.821885 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:00.824301 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:00.825701 5378 hybrid_clock.cc:648] HybridClock initialized: now 1736542620825675 us; error 41 us; skew 500 ppm
I20250110 20:57:00.826443 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:00.833004 5378 webserver.cc:458] Webserver started at http://127.4.171.195:44823/ using document root <none> and password file <none>
I20250110 20:57:00.834188 5378 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:00.834414 5378 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:57:00.834940 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:57:00.839767 5378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data/instance:
uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
format_stamp: "Formatted at 2025-01-10 20:57:00 on dist-test-slave-98kh"
I20250110 20:57:00.840881 5378 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/instance:
uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
format_stamp: "Formatted at 2025-01-10 20:57:00 on dist-test-slave-98kh"
I20250110 20:57:00.847699 5378 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250110 20:57:00.853267 5395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:00.854274 5378 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250110 20:57:00.854557 5378 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal
uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
format_stamp: "Formatted at 2025-01-10 20:57:00 on dist-test-slave-98kh"
I20250110 20:57:00.854866 5378 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:00.901392 5378 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:00.902729 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:00.903208 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:00.906468 5378 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:57:00.911468 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:57:00.911666 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:00.911906 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:57:00.912052 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:00.936393 5417 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:57:00.937001 5417 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:00.938975 5417 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:57:00.940569 4835 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:57:00.942103 4989 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:57:00.944509 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b84772eba3f0460faf8523f76eb57e53; no voters: 14d1e2d5c72f498f915c367e8750361f, 63aa77ee6ec84a6e82132b2de9e77486
I20250110 20:57:00.945397 5417 raft_consensus.cc:2743] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:01.046835 5378 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.195:32843
I20250110 20:57:01.046924 5508 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.195:32843 every 8 connection(s)
I20250110 20:57:01.049153 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/data/info.pb
I20250110 20:57:01.054540 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 5378
I20250110 20:57:01.055049 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/instance
I20250110 20:57:01.083642 5509 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:57:01.084100 5509 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:01.085088 5509 heartbeater.cc:510] Master 127.4.171.252:37233 requested a full tablet report, sending...
I20250110 20:57:01.086817 5510 heartbeater.cc:346] Connected to a master server at 127.4.171.253:34381
I20250110 20:57:01.087131 5510 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:01.087477 4979 ts_manager.cc:194] Registered new tserver with Master: 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:01.088797 5510 heartbeater.cc:510] Master 127.4.171.253:34381 requested a full tablet report, sending...
I20250110 20:57:01.089066 4979 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.171.195:36467
I20250110 20:57:01.090132 5511 heartbeater.cc:346] Connected to a master server at 127.4.171.254:34617
I20250110 20:57:01.090462 5511 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:01.091251 5511 heartbeater.cc:510] Master 127.4.171.254:34617 requested a full tablet report, sending...
I20250110 20:57:01.091253 4900 ts_manager.cc:194] Registered new tserver with Master: 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:01.094576 4825 ts_manager.cc:194] Registered new tserver with Master: 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:01.099895 4783 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:57:01.151902 4979 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32834:
name: "default.Table_74c612e2a24642b6824ee1693a2c242b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:01.151847 4978 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32834:
name: "default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:01.154943 4979 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74c612e2a24642b6824ee1693a2c242b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:01.155700 4976 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32834:
name: "default.Table_0b176fb73c8e4d22886b3593471638da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:01.157313 4976 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b176fb73c8e4d22886b3593471638da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:01.158515 4977 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:32834:
name: "default.Table_ff58f7ae21d3423dac4418eaa9323d72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:01.160537 4977 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff58f7ae21d3423dac4418eaa9323d72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:01.154968 4978 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:01.216822 4979 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:32834:
table { table_name: "default.Table_74c612e2a24642b6824ee1693a2c242b" } new_table_name: "default.Table_8b67c126d29d4216894b36f2c65e6b4d" modify_external_catalogs: true
I20250110 20:57:01.262487 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 4937
W20250110 20:57:01.291292 4809 connection.cc:537] server connection from 127.0.0.1:42210 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:01.292105 4887 connection.cc:537] server connection from 127.0.0.1:59918 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:01.380954 5368 heartbeater.cc:643] Failed to heartbeat to 127.4.171.252:37233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:57:01.592794 5509 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
W20250110 20:57:01.596910 5509 heartbeater.cc:643] Failed to heartbeat to 127.4.171.252:37233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
W20250110 20:57:01.643328 5222 heartbeater.cc:643] Failed to heartbeat to 127.4.171.252:37233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:57:01.732878 5560 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:57:01.733325 5560 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:01.734828 5560 leader_election.cc:290] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:57:01.735827 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
W20250110 20:57:01.738188 4812 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20250110 20:57:01.741911 4812 leader_election.cc:336] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233): Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:57:01.742383 4812 leader_election.cc:304] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486; no voters: 14d1e2d5c72f498f915c367e8750361f, b84772eba3f0460faf8523f76eb57e53
I20250110 20:57:01.742926 5560 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:01.763998 5563 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
I20250110 20:57:01.764350 5563 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:01.765811 5563 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:57:01.766747 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:57:01.767329 4834 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b84772eba3f0460faf8523f76eb57e53 in term 1.
I20250110 20:57:01.768281 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486, b84772eba3f0460faf8523f76eb57e53; no voters:
I20250110 20:57:01.769040 5563 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:57:01.769423 5563 raft_consensus.cc:491] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 14d1e2d5c72f498f915c367e8750361f)
W20250110 20:57:01.769533 4887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250110 20:57:01.769802 5563 raft_consensus.cc:3054] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 1 FOLLOWER]: Advancing to term 2
W20250110 20:57:01.772981 4887 leader_election.cc:336] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233): Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:57:01.779444 5563 raft_consensus.cc:513] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:01.781394 5563 leader_election.cc:290] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 election: Requested vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:57:01.782196 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b84772eba3f0460faf8523f76eb57e53" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486"
I20250110 20:57:01.782604 4834 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 1 FOLLOWER]: Advancing to term 2
W20250110 20:57:01.785061 4887 leader_election.cc:336] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233): Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111)
I20250110 20:57:01.788713 4834 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b84772eba3f0460faf8523f76eb57e53 in term 2.
I20250110 20:57:01.789649 4887 leader_election.cc:304] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486, b84772eba3f0460faf8523f76eb57e53; no voters: 14d1e2d5c72f498f915c367e8750361f
I20250110 20:57:01.790385 5563 raft_consensus.cc:2798] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:57:01.791286 5563 raft_consensus.cc:695] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [term 2 LEADER]: Becoming Leader. State: Replica: b84772eba3f0460faf8523f76eb57e53, State: Running, Role: LEADER
I20250110 20:57:01.792065 5563 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:01.795076 5566 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84772eba3f0460faf8523f76eb57e53. Latest consensus state: current_term: 2 leader_uuid: "b84772eba3f0460faf8523f76eb57e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:01.795611 5566 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:02.282498 4834 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 2 FOLLOWER]: Refusing update from remote peer b84772eba3f0460faf8523f76eb57e53: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250110 20:57:02.283756 5566 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20250110 20:57:02.288654 5571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84772eba3f0460faf8523f76eb57e53. Latest consensus state: current_term: 2 leader_uuid: "b84772eba3f0460faf8523f76eb57e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:02.289543 5571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:02.295548 5563 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b84772eba3f0460faf8523f76eb57e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:02.296247 5563 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:02.298718 5571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b84772eba3f0460faf8523f76eb57e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:02.299384 5571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [sys.catalog]: This master's current role is: FOLLOWER
W20250110 20:57:02.300993 4887 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 -> Peer 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233): Couldn't send request to peer 14d1e2d5c72f498f915c367e8750361f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:02.313575 5567 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:57:02.321697 5567 catalog_manager.cc:670] Loaded metadata for table default.Table_74c612e2a24642b6824ee1693a2c242b [id=88781430b6ec4c1b9663bc6544fe95b4]
I20250110 20:57:02.323190 5567 catalog_manager.cc:670] Loaded metadata for table default.Table_0b176fb73c8e4d22886b3593471638da [id=896c7b90b1bd4d048082ccac12d70301]
I20250110 20:57:02.324568 5567 catalog_manager.cc:670] Loaded metadata for table default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]
I20250110 20:57:02.325985 5567 catalog_manager.cc:670] Loaded metadata for table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]
I20250110 20:57:02.332105 5567 tablet_loader.cc:96] loaded metadata for tablet 5c2e7b5f8e6440ba8fb8890132781449 (table default.Table_0b176fb73c8e4d22886b3593471638da [id=896c7b90b1bd4d048082ccac12d70301])
I20250110 20:57:02.333042 5567 tablet_loader.cc:96] loaded metadata for tablet 61d385de448243d4b4c5f76c8d4967af (table default.Table_74c612e2a24642b6824ee1693a2c242b [id=88781430b6ec4c1b9663bc6544fe95b4])
I20250110 20:57:02.333840 5567 tablet_loader.cc:96] loaded metadata for tablet a04595ff7d1e4402a486f4ddf420ea29 (table default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec])
I20250110 20:57:02.334506 5567 tablet_loader.cc:96] loaded metadata for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb (table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f])
I20250110 20:57:02.335350 5567 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:57:02.338239 5567 catalog_manager.cc:1260] Loaded cluster ID: 9189b87cd4594c99bd9b29530bf32da1
I20250110 20:57:02.338405 5567 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:57:02.341697 5567 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:57:02.344260 5567 catalog_manager.cc:5910] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Loaded TSK: 0
I20250110 20:57:02.345157 5567 catalog_manager.cc:1515] Initializing in-progress tserver states...
W20250110 20:57:02.397536 4788 debug-util.cc:398] Leaking SignalData structure 0x7b080002c080 after lost signal to thread 5522
W20250110 20:57:01.292568 5522 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.171.252:37233)
W20250110 20:57:02.474232 5522 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.AlterTable -> {remote=127.4.171.252:37233, user_credentials={real_user=slave}} blocked reactor thread for 1.18313e+06us
I20250110 20:57:02.475591 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.252:37233
--webserver_interface=127.4.171.252
--webserver_port=45215
--builtin_ntp_servers=127.4.171.212:38225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250110 20:57:02.475834 5522 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.171.252:37233)
W20250110 20:57:02.480839 5522 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.171.252:37233)
I20250110 20:57:02.490680 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_74c612e2a24642b6824ee1693a2c242b" } new_table_name: "default.Table_8b67c126d29d4216894b36f2c65e6b4d" modify_external_catalogs: true
I20250110 20:57:02.492190 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ff58f7ae21d3423dac4418eaa9323d72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:02.491431 4898 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_0b176fb73c8e4d22886b3593471638da" } new_table_name: "default.Table_2c5da9973d434c468403ff4ea730540b" modify_external_catalogs: true
I20250110 20:57:02.500646 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_831f721453eb4bba938691fdb29aa937"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:02.493187 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:02.524091 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff58f7ae21d3423dac4418eaa9323d72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:02.524547 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:02.499583 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_14adb01640aa40d699cfb94980d7aea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:02.526137 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_831f721453eb4bba938691fdb29aa937 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:02.527036 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14adb01640aa40d699cfb94980d7aea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:02.531584 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_99d6db4f69ff4911a23faa22b2842fb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:02.531952 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d478a83978624ca7b2b0c3af95388ff2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:02.533286 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99d6db4f69ff4911a23faa22b2842fb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:02.534510 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d478a83978624ca7b2b0c3af95388ff2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:02.541333 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61d385de448243d4b4c5f76c8d4967af consensus state has no leader
I20250110 20:57:02.566047 4894 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_831f721453eb4bba938691fdb29aa937" } modify_external_catalogs: true
I20250110 20:57:02.566717 4894 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_831f721453eb4bba938691fdb29aa937" } modify_external_catalogs: true
I20250110 20:57:02.571002 4900 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8b67c126d29d4216894b36f2c65e6b4d" } modify_external_catalogs: true
I20250110 20:57:02.575026 4900 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8b67c126d29d4216894b36f2c65e6b4d" } modify_external_catalogs: true
W20250110 20:57:02.582505 4898 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c2e7b5f8e6440ba8fb8890132781449 has no consensus state
W20250110 20:57:02.597879 4886 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 61d385de448243d4b4c5f76c8d4967af (table default.Table_8b67c126d29d4216894b36f2c65e6b4d [id=88781430b6ec4c1b9663bc6544fe95b4], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 61d385de448243d4b4c5f76c8d4967af consensus state has no leader
I20250110 20:57:02.639214 4894 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet cdead81e36b04c09afabf55c64998a97
I20250110 20:57:02.646845 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 61d385de448243d4b4c5f76c8d4967af
I20250110 20:57:02.680063 5444 tablet_service.cc:1467] Processing CreateTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.685261 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 (DEFAULT_TABLE table=default.Table_2c5da9973d434c468403ff4ea730540b [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.681457 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d385de448243d4b4c5f76c8d4967af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:02.687007 5442 tablet_service.cc:1467] Processing CreateTablet for tablet a04595ff7d1e4402a486f4ddf420ea29 (DEFAULT_TABLE table=default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.696501 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.697389 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2e7b5f8e6440ba8fb8890132781449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.699972 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04595ff7d1e4402a486f4ddf420ea29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.694718 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 (DEFAULT_TABLE table=default.Table_2c5da9973d434c468403ff4ea730540b [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.703270 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2e7b5f8e6440ba8fb8890132781449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.696543 5154 tablet_service.cc:1467] Processing CreateTablet for tablet a04595ff7d1e4402a486f4ddf420ea29 (DEFAULT_TABLE table=default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.706063 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04595ff7d1e4402a486f4ddf420ea29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.695415 5156 tablet_service.cc:1467] Processing CreateTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.710431 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.693488 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d385de448243d4b4c5f76c8d4967af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:02.723476 5303 tablet_service.cc:1467] Processing CreateTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.733597 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.731931 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 (DEFAULT_TABLE table=default.Table_2c5da9973d434c468403ff4ea730540b [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.730481 5301 tablet_service.cc:1467] Processing CreateTablet for tablet a04595ff7d1e4402a486f4ddf420ea29 (DEFAULT_TABLE table=default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.735875 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c2e7b5f8e6440ba8fb8890132781449. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.736445 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04595ff7d1e4402a486f4ddf420ea29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.725106 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet 61d385de448243d4b4c5f76c8d4967af with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:02.742722 5594 tablet_bootstrap.cc:492] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:02.749738 5594 tablet_bootstrap.cc:654] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:02.751998 5594 log.cc:826] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Log is configured to *not* fsync() on all Append() calls
I20250110 20:57:02.756661 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d478a83978624ca7b2b0c3af95388ff2" } new_table_name: "default.Table_e8fd956943c647e1890ffe3f3d3597f7" modify_external_catalogs: true
I20250110 20:57:02.760154 5152 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd2cc66ab594f849f142d7847a1e0f5 (DEFAULT_TABLE table=default.Table_14adb01640aa40d699cfb94980d7aea3 [id=1c2a6b50e68e4363b5ebf736451866a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.761317 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd2cc66ab594f849f142d7847a1e0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.762230 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_be169b281c0042f28eb8b1c32aa79869"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:02.763912 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be169b281c0042f28eb8b1c32aa79869 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:02.769258 5594 tablet_bootstrap.cc:492] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:02.769762 5594 ts_tablet_manager.cc:1397] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.005s
W20250110 20:57:02.782001 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80a22306f7c7487d996969e349f09766 consensus state has no leader
I20250110 20:57:02.785703 5299 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd2cc66ab594f849f142d7847a1e0f5 (DEFAULT_TABLE table=default.Table_14adb01640aa40d699cfb94980d7aea3 [id=1c2a6b50e68e4363b5ebf736451866a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.786864 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd2cc66ab594f849f142d7847a1e0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.790762 5597 tablet_bootstrap.cc:492] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:02.793547 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2c5da9973d434c468403ff4ea730540b" } new_table_name: "default.Table_54d6bd0b8362483ba4440d3d954c1383" modify_external_catalogs: true
I20250110 20:57:02.802064 4899 master_service.cc:946] ReplaceTablet: received request to replace tablet 5c2e7b5f8e6440ba8fb8890132781449 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:02.803588 4898 master_service.cc:946] ReplaceTablet: received request to replace tablet 5c2e7b5f8e6440ba8fb8890132781449 from {username='slave'} at 127.0.0.1:36568
W20250110 20:57:02.805331 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 61d385de448243d4b4c5f76c8d4967af because the tablet was not found. No further retry: Not found: Tablet not found: 61d385de448243d4b4c5f76c8d4967af
I20250110 20:57:02.806011 5597 tablet_bootstrap.cc:654] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:02.806321 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_fc4b2c0b84794d9aaa7a4c722b38ef8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:02.806946 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 80a22306f7c7487d996969e349f09766 (DEFAULT_TABLE table=default.Table_e8fd956943c647e1890ffe3f3d3597f7 [id=3fb5402eb3a944d989a916f70a71c2d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.807932 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a22306f7c7487d996969e349f09766. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:02.807857 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc4b2c0b84794d9aaa7a4c722b38ef8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:02.808274 5597 log.cc:826] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Log is configured to *not* fsync() on all Append() calls
I20250110 20:57:02.808470 5298 tablet_service.cc:1467] Processing CreateTablet for tablet 80a22306f7c7487d996969e349f09766 (DEFAULT_TABLE table=default.Table_e8fd956943c647e1890ffe3f3d3597f7 [id=3fb5402eb3a944d989a916f70a71c2d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.809554 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_0fb62c9c09f741c7beea780971c66f3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:02.810034 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a22306f7c7487d996969e349f09766. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:02.811157 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fb62c9c09f741c7beea780971c66f3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:02.811133 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 0fd2cc66ab594f849f142d7847a1e0f5 (DEFAULT_TABLE table=default.Table_14adb01640aa40d699cfb94980d7aea3 [id=1c2a6b50e68e4363b5ebf736451866a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.812116 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd2cc66ab594f849f142d7847a1e0f5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:02.827387 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 61d385de448243d4b4c5f76c8d4967af because the tablet was not found. No further retry: Not found: Tablet not found: 61d385de448243d4b4c5f76c8d4967af
I20250110 20:57:02.829051 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 80a22306f7c7487d996969e349f09766 (DEFAULT_TABLE table=default.Table_e8fd956943c647e1890ffe3f3d3597f7 [id=3fb5402eb3a944d989a916f70a71c2d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.830215 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80a22306f7c7487d996969e349f09766. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:02.842916 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 61d385de448243d4b4c5f76c8d4967af because the tablet was not found. No further retry: Not found: Tablet not found: 61d385de448243d4b4c5f76c8d4967af
I20250110 20:57:02.849524 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 (DEFAULT_TABLE table=default.Table_99d6db4f69ff4911a23faa22b2842fb6 [id=7a14b963398847d6b19313baae9772ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.850553 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7680441899e14a2d9a6ff45d0e0d69b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.850687 4895 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_be169b281c0042f28eb8b1c32aa79869" } modify_external_catalogs: true
I20250110 20:57:02.851284 4895 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_be169b281c0042f28eb8b1c32aa79869" } modify_external_catalogs: true
W20250110 20:57:02.852510 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c2e7b5f8e6440ba8fb8890132781449 consensus state has no leader
I20250110 20:57:02.855377 4900 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e8fd956943c647e1890ffe3f3d3597f7" } modify_external_catalogs: true
I20250110 20:57:02.855978 4900 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e8fd956943c647e1890ffe3f3d3597f7" } modify_external_catalogs: true
I20250110 20:57:02.860786 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_45d415fe81e34e7a8001662d4d519278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:02.865130 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 (DEFAULT_TABLE table=default.Table_99d6db4f69ff4911a23faa22b2842fb6 [id=7a14b963398847d6b19313baae9772ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.866322 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7680441899e14a2d9a6ff45d0e0d69b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.870807 5600 tablet_bootstrap.cc:492] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:02.876713 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 (DEFAULT_TABLE table=default.Table_99d6db4f69ff4911a23faa22b2842fb6 [id=7a14b963398847d6b19313baae9772ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:02.877789 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7680441899e14a2d9a6ff45d0e0d69b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:02.882612 5594 raft_consensus.cc:357] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:02.883505 5594 raft_consensus.cc:383] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:02.883785 5594 raft_consensus.cc:738] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:02.884877 5594 consensus_queue.cc:260] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:02.886410 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45d415fe81e34e7a8001662d4d519278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:02.886878 5600 tablet_bootstrap.cc:654] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:02.888240 5369 heartbeater.cc:502] Master 127.4.171.253:34381 was elected leader, sending a full tablet report...
I20250110 20:57:02.898129 4898 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 5c2e7b5f8e6440ba8fb8890132781449
I20250110 20:57:02.899549 4898 catalog_manager.cc:6596] ReplaceTablet: tablet 5c2e7b5f8e6440ba8fb8890132781449 deleted and replaced by tablet 637dda10276947d5842ed24bbf044d0d
I20250110 20:57:02.902122 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:02.903241 5597 tablet_bootstrap.cc:492] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:02.903753 5597 ts_tablet_manager.cc:1397] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.114s user 0.012s sys 0.040s
W20250110 20:57:02.903704 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 5c2e7b5f8e6440ba8fb8890132781449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c2e7b5f8e6440ba8fb8890132781449 already in progress: creating tablet
I20250110 20:57:02.908792 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:02.917680 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:02.922598 5600 log.cc:826] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Log is configured to *not* fsync() on all Append() calls
W20250110 20:57:02.925623 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 5c2e7b5f8e6440ba8fb8890132781449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c2e7b5f8e6440ba8fb8890132781449 already in progress: creating tablet
W20250110 20:57:02.931335 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 5c2e7b5f8e6440ba8fb8890132781449 because the tablet was not found. No further retry: Not found: Tablet not found: 5c2e7b5f8e6440ba8fb8890132781449
I20250110 20:57:02.933195 5597 raft_consensus.cc:357] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:02.934139 5597 raft_consensus.cc:383] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:02.934438 5597 raft_consensus.cc:738] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:02.935392 5597 consensus_queue.cc:260] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:02.953166 5597 ts_tablet_manager.cc:1428] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.049s user 0.031s sys 0.010s
I20250110 20:57:02.954959 5600 tablet_bootstrap.cc:492] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:02.955451 5600 ts_tablet_manager.cc:1397] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.002s
I20250110 20:57:02.956822 5594 ts_tablet_manager.cc:1428] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.187s user 0.031s sys 0.004s
I20250110 20:57:02.957752 5597 tablet_bootstrap.cc:492] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:02.953701 5223 heartbeater.cc:502] Master 127.4.171.253:34381 was elected leader, sending a full tablet report...
I20250110 20:57:02.962347 5510 heartbeater.cc:502] Master 127.4.171.253:34381 was elected leader, sending a full tablet report...
I20250110 20:57:02.969286 5597 tablet_bootstrap.cc:654] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:02.970968 5594 tablet_bootstrap.cc:492] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:02.978583 4895 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 44fb3dd80dcc4b478a6ba05dcd3675b1
I20250110 20:57:02.985217 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_99d6db4f69ff4911a23faa22b2842fb6" } new_table_name: "default.Table_6c07fcebfdf6402a8876074a13c5708b" modify_external_catalogs: true
I20250110 20:57:02.999835 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 80a22306f7c7487d996969e349f09766
I20250110 20:57:03.000509 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_4a4edbef0d0942e6ac9ceb7600dab9d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:03.002089 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a4edbef0d0942e6ac9ceb7600dab9d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.003479 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:03.006057 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 80a22306f7c7487d996969e349f09766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a22306f7c7487d996969e349f09766 already in progress: creating tablet
I20250110 20:57:03.007575 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:03.009029 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 80a22306f7c7487d996969e349f09766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a22306f7c7487d996969e349f09766 already in progress: creating tablet
I20250110 20:57:03.010032 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:03.011574 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 80a22306f7c7487d996969e349f09766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a22306f7c7487d996969e349f09766 already in progress: creating tablet
W20250110 20:57:03.014055 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7680441899e14a2d9a6ff45d0e0d69b8 consensus state has no leader
I20250110 20:57:03.015949 5597 tablet_bootstrap.cc:492] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.016357 5597 ts_tablet_manager.cc:1397] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.061s user 0.017s sys 0.007s
I20250110 20:57:03.017314 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:03.018867 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:03.019028 5597 raft_consensus.cc:357] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.019692 5593 tablet_replica.cc:331] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:03.019749 5597 raft_consensus.cc:383] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.020094 5597 raft_consensus.cc:738] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.020542 5593 raft_consensus.cc:2238] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.021098 5593 raft_consensus.cc:2267] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.021466 5594 tablet_bootstrap.cc:654] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:03.021886 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 80a22306f7c7487d996969e349f09766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a22306f7c7487d996969e349f09766 already in progress: creating tablet
I20250110 20:57:03.024626 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:03.024853 5600 raft_consensus.cc:357] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.025794 5600 raft_consensus.cc:383] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.026098 5600 raft_consensus.cc:738] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:03.026137 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 5c2e7b5f8e6440ba8fb8890132781449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c2e7b5f8e6440ba8fb8890132781449 already in progress: creating tablet
I20250110 20:57:03.027014 5600 consensus_queue.cc:260] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.028959 5593 ts_tablet_manager.cc:1905] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.029258 5597 consensus_queue.cc:260] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.031330 5597 ts_tablet_manager.cc:1428] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250110 20:57:03.032169 5597 tablet_bootstrap.cc:492] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:03.035030 5225 tablet.cc:2367] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:57:03.035781 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:03.037371 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 80a22306f7c7487d996969e349f09766 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80a22306f7c7487d996969e349f09766 already in progress: creating tablet
I20250110 20:57:03.040176 5597 tablet_bootstrap.cc:654] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.058175 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_45d415fe81e34e7a8001662d4d519278" } new_table_name: "default.Table_45e30e66a68c4189ab430736f6f301be" modify_external_catalogs: true
W20250110 20:57:03.074971 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10c96d0d100d4f7eb4dcbf531c66b65e has no consensus state
W20250110 20:57:03.076393 5514 tablet.cc:2367] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:57:03.084146 5606 raft_consensus.cc:491] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.084650 5606 raft_consensus.cc:513] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.085887 5600 ts_tablet_manager.cc:1428] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.130s user 0.049s sys 0.010s
I20250110 20:57:03.091100 5606 leader_election.cc:290] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:03.098670 5600 tablet_bootstrap.cc:492] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:03.099983 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca (DEFAULT_TABLE table=default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.100237 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca (DEFAULT_TABLE table=default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.101053 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332dfe3a0e45468695aeb7d7eb7c30ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.101125 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332dfe3a0e45468695aeb7d7eb7c30ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.105109 5597 tablet_bootstrap.cc:492] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.105535 5597 ts_tablet_manager.cc:1397] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.017s
I20250110 20:57:03.107385 5594 tablet_bootstrap.cc:492] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:03.107860 5594 ts_tablet_manager.cc:1397] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.137s user 0.013s sys 0.018s
I20250110 20:57:03.107884 5597 raft_consensus.cc:357] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.108482 5597 raft_consensus.cc:383] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:03.108608 5375 tablet.cc:2367] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:57:03.108734 5597 raft_consensus.cc:738] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.108685 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd2cc66ab594f849f142d7847a1e0f5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:03.110687 5110 leader_election.cc:343] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.114629 5600 tablet_bootstrap.cc:654] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.114358 5594 raft_consensus.cc:357] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.115041 5594 raft_consensus.cc:383] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.115316 5594 raft_consensus.cc:738] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.109313 5597 consensus_queue.cc:260] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.120770 5594 consensus_queue.cc:260] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.127123 5597 ts_tablet_manager.cc:1428] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250110 20:57:03.128001 5597 tablet_bootstrap.cc:492] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.130321 4895 master_service.cc:946] ReplaceTablet: received request to replace tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:03.132124 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ccaf097e265d4ed6a3ed2e9c0984229a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:03.133746 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccaf097e265d4ed6a3ed2e9c0984229a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.135089 5597 tablet_bootstrap.cc:654] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.135787 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet 80a22306f7c7487d996969e349f09766 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:03.137957 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:03.140631 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 5c2e7b5f8e6440ba8fb8890132781449 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c2e7b5f8e6440ba8fb8890132781449 already in progress: creating tablet
I20250110 20:57:03.142243 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:03.143776 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_19bc099f0c20410484ad566009bebba3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:03.146260 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19bc099f0c20410484ad566009bebba3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.150230 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd2cc66ab594f849f142d7847a1e0f5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:03.151226 5323 raft_consensus.cc:2463] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:03.152721 5111 leader_election.cc:304] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:03.154805 5606 raft_consensus.cc:2798] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:03.155149 5606 raft_consensus.cc:491] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:03.155426 5606 raft_consensus.cc:3054] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:03.161904 5606 raft_consensus.cc:513] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.164971 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd2cc66ab594f849f142d7847a1e0f5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:03.165589 5323 raft_consensus.cc:3054] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:03.172994 5323 raft_consensus.cc:2463] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:03.173753 5606 leader_election.cc:290] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:03.174739 5111 leader_election.cc:304] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:03.178906 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd2cc66ab594f849f142d7847a1e0f5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
W20250110 20:57:03.180078 5110 leader_election.cc:343] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.181603 5600 tablet_bootstrap.cc:492] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:03.182425 5606 raft_consensus.cc:2798] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:03.183334 4899 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_45e30e66a68c4189ab430736f6f301be" } modify_external_catalogs: true
I20250110 20:57:03.183940 4899 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_45e30e66a68c4189ab430736f6f301be" } modify_external_catalogs: true
I20250110 20:57:03.184671 5600 ts_tablet_manager.cc:1397] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.086s user 0.017s sys 0.003s
I20250110 20:57:03.183363 5606 raft_consensus.cc:695] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:03.189617 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb
I20250110 20:57:03.190903 4895 catalog_manager.cc:6596] ReplaceTablet: tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb deleted and replaced by tablet 01962c5b7ae64f80b998948d18250b4c
I20250110 20:57:03.148836 5597 tablet_bootstrap.cc:492] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.191665 5597 ts_tablet_manager.cc:1397] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.004s
I20250110 20:57:03.193751 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb with delete_type TABLET_DATA_DELETED (replaced by tablet 01962c5b7ae64f80b998948d18250b4c) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:03.186417 5606 consensus_queue.cc:237] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:03.196573 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb already in progress: creating tablet
I20250110 20:57:03.199292 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb with delete_type TABLET_DATA_DELETED (replaced by tablet 01962c5b7ae64f80b998948d18250b4c) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:03.200199 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb with delete_type TABLET_DATA_DELETED (replaced by tablet 01962c5b7ae64f80b998948d18250b4c) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:03.200774 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb already in progress: creating tablet
I20250110 20:57:03.206445 5597 raft_consensus.cc:357] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.207150 5597 raft_consensus.cc:383] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.207394 5597 raft_consensus.cc:738] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.208004 5597 consensus_queue.cc:260] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.204514 5600 raft_consensus.cc:357] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.209079 5600 raft_consensus.cc:383] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.209353 5600 raft_consensus.cc:738] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.209988 5600 consensus_queue.cc:260] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.211952 5600 ts_tablet_manager.cc:1428] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.025s user 0.000s sys 0.003s
I20250110 20:57:03.212929 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_4a4edbef0d0942e6ac9ceb7600dab9d1" } modify_external_catalogs: true
I20250110 20:57:03.213467 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_4a4edbef0d0942e6ac9ceb7600dab9d1" } modify_external_catalogs: true
W20250110 20:57:03.214191 5581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:03.214928 5581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:03.215590 5581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250110 20:57:03.218361 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_fc4b2c0b84794d9aaa7a4c722b38ef8f" } new_table_name: "default.Table_2fee2488f5df4696853086b480b0a2e8" modify_external_catalogs: true
I20250110 20:57:03.222174 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_aacabce3495d4b85b5f97043bc6a7242"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:03.223687 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aacabce3495d4b85b5f97043bc6a7242 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.225219 5597 ts_tablet_manager.cc:1428] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250110 20:57:03.226171 5600 tablet_bootstrap.cc:492] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:03.228443 5597 tablet_bootstrap.cc:492] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.234179 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_cd3dd0abc91e4209b6f8586377dafc01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:03.236011 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd3dd0abc91e4209b6f8586377dafc01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.241596 5597 tablet_bootstrap.cc:654] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.247123 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc0d3abef5741339fdbb9abeed20b15 (DEFAULT_TABLE table=default.Table_fc4b2c0b84794d9aaa7a4c722b38ef8f [id=9daaee4eb9574713b7914a551680c52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.248162 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc0d3abef5741339fdbb9abeed20b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.245180 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca (DEFAULT_TABLE table=default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.251751 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 332dfe3a0e45468695aeb7d7eb7c30ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.257236 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc0d3abef5741339fdbb9abeed20b15 (DEFAULT_TABLE table=default.Table_fc4b2c0b84794d9aaa7a4c722b38ef8f [id=9daaee4eb9574713b7914a551680c52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.258230 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc0d3abef5741339fdbb9abeed20b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.264154 5600 tablet_bootstrap.cc:654] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.271276 5594 ts_tablet_manager.cc:1428] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.163s user 0.003s sys 0.004s
I20250110 20:57:03.272555 5594 tablet_bootstrap.cc:492] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:03.273722 5618 raft_consensus.cc:491] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.274163 5618 raft_consensus.cc:513] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.275892 5618 leader_election.cc:290] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:03.277305 5601 raft_consensus.cc:491] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.277915 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a22306f7c7487d996969e349f09766" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:03.277848 5601 raft_consensus.cc:513] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:03.279325 5111 leader_election.cc:343] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.283857 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80a22306f7c7487d996969e349f09766" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:03.285099 5110 leader_election.cc:343] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.285589 5110 leader_election.cc:304] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:03.286721 5618 raft_consensus.cc:2743] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:03.291597 5600 tablet_bootstrap.cc:492] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:03.292071 5600 ts_tablet_manager.cc:1397] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.006s
I20250110 20:57:03.292492 5601 leader_election.cc:290] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:03.295786 5600 raft_consensus.cc:357] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.296447 5600 raft_consensus.cc:383] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.296692 5600 raft_consensus.cc:738] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.297396 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 2dc0d3abef5741339fdbb9abeed20b15 (DEFAULT_TABLE table=default.Table_fc4b2c0b84794d9aaa7a4c722b38ef8f [id=9daaee4eb9574713b7914a551680c52b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.297720 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 637dda10276947d5842ed24bbf044d0d (DEFAULT_TABLE table=default.Table_54d6bd0b8362483ba4440d3d954c1383 [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.298462 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dc0d3abef5741339fdbb9abeed20b15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.298730 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637dda10276947d5842ed24bbf044d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.299077 5597 tablet_bootstrap.cc:492] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.299463 5597 ts_tablet_manager.cc:1397] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.071s user 0.012s sys 0.002s
I20250110 20:57:03.300385 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 637dda10276947d5842ed24bbf044d0d (DEFAULT_TABLE table=default.Table_54d6bd0b8362483ba4440d3d954c1383 [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.301268 5593 ts_tablet_manager.cc:1918] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.301719 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637dda10276947d5842ed24bbf044d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.301764 5593 log.cc:1198] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/5c2e7b5f8e6440ba8fb8890132781449
I20250110 20:57:03.302337 5597 raft_consensus.cc:357] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.305685 5600 consensus_queue.cc:260] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.306504 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 637dda10276947d5842ed24bbf044d0d (DEFAULT_TABLE table=default.Table_54d6bd0b8362483ba4440d3d954c1383 [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.307516 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637dda10276947d5842ed24bbf044d0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.309834 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04595ff7d1e4402a486f4ddf420ea29" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:03.303038 5597 raft_consensus.cc:383] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.310407 5323 raft_consensus.cc:2463] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:03.314791 5398 leader_election.cc:304] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:03.316927 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_19bc099f0c20410484ad566009bebba3" } new_table_name: "default.Table_b115c55a632e444795c1dfc779b62d48" modify_external_catalogs: true
I20250110 20:57:03.322256 5601 raft_consensus.cc:2798] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:03.322690 5601 raft_consensus.cc:491] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:03.323344 5601 raft_consensus.cc:3054] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:03.326195 4899 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 10c96d0d100d4f7eb4dcbf531c66b65e
I20250110 20:57:03.310503 5597 raft_consensus.cc:738] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.331615 5600 ts_tablet_manager.cc:1428] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.039s user 0.000s sys 0.006s
I20250110 20:57:03.302749 5593 ts_tablet_manager.cc:1939] T 5c2e7b5f8e6440ba8fb8890132781449 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:03.334708 5593 tablet_replica.cc:331] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:03.335038 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 5c2e7b5f8e6440ba8fb8890132781449 (table default.Table_54d6bd0b8362483ba4440d3d954c1383 [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
I20250110 20:57:03.335251 5593 raft_consensus.cc:2238] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.335693 5593 raft_consensus.cc:2267] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.335747 4896 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet ff8c8b7dc1bc4466805a601dc00074be
I20250110 20:57:03.336362 5597 consensus_queue.cc:260] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.339721 5597 ts_tablet_manager.cc:1428] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20250110 20:57:03.340663 5597 tablet_bootstrap.cc:492] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.337421 5601 raft_consensus.cc:513] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.349077 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04595ff7d1e4402a486f4ddf420ea29" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:03.349628 5323 raft_consensus.cc:3054] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:03.350282 5600 tablet_bootstrap.cc:492] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:03.357868 5323 raft_consensus.cc:2463] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:03.359196 5398 leader_election.cc:304] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
W20250110 20:57:03.360904 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dc0d3abef5741339fdbb9abeed20b15 consensus state has no leader
I20250110 20:57:03.365100 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:03.366003 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04595ff7d1e4402a486f4ddf420ea29" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:03.366433 5595 tablet_replica.cc:331] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:03.367249 5595 raft_consensus.cc:2238] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.367316 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a04595ff7d1e4402a486f4ddf420ea29" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:03.367774 5595 raft_consensus.cc:2267] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.370205 5595 ts_tablet_manager.cc:1905] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.368148 5176 raft_consensus.cc:2463] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:03.372471 5601 leader_election.cc:290] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:03.372794 5600 tablet_bootstrap.cc:654] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.372730 5624 raft_consensus.cc:2798] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:03.373313 5624 raft_consensus.cc:695] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:03.380283 5593 ts_tablet_manager.cc:1905] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.381592 5594 tablet_bootstrap.cc:654] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.380789 5624 consensus_queue.cc:237] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:03.385798 5581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250110 20:57:03.386394 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:03.387166 5592 tablet_replica.cc:331] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:03.387773 5597 tablet_bootstrap.cc:654] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:03.386176 5581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:03.389379 5581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:57:03.389744 5581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:57:03.428802 4897 master_service.cc:946] ReplaceTablet: received request to replace tablet 332dfe3a0e45468695aeb7d7eb7c30ca from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:03.430958 5597 tablet_bootstrap.cc:492] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.431594 5597 ts_tablet_manager.cc:1397] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.004s
I20250110 20:57:03.431440 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_213a26b346b94108a30a80c719397f12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:03.434836 5592 raft_consensus.cc:2238] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.434404 5597 raft_consensus.cc:357] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.435385 5592 raft_consensus.cc:2267] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.435799 5597 raft_consensus.cc:383] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.436232 5597 raft_consensus.cc:738] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.436863 5597 consensus_queue.cc:260] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.439196 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet 332dfe3a0e45468695aeb7d7eb7c30ca from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:03.439368 5597 ts_tablet_manager.cc:1428] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:57:03.440531 5606 raft_consensus.cc:491] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.440937 5606 raft_consensus.cc:513] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.442195 5592 ts_tablet_manager.cc:1905] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.442636 5606 leader_election.cc:290] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:03.440357 5597 tablet_bootstrap.cc:492] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.443774 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7680441899e14a2d9a6ff45d0e0d69b8" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:03.445219 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7680441899e14a2d9a6ff45d0e0d69b8" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:03.446231 5593 ts_tablet_manager.cc:1918] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.446622 5593 log.cc:1198] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/80a22306f7c7487d996969e349f09766
W20250110 20:57:03.447394 5110 leader_election.cc:343] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.448325 5600 tablet_bootstrap.cc:492] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:03.448748 5600 ts_tablet_manager.cc:1397] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.009s
W20250110 20:57:03.449568 5111 leader_election.cc:343] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.449752 5597 tablet_bootstrap.cc:654] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.449970 5111 leader_election.cc:304] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:03.450644 5606 raft_consensus.cc:2743] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:03.451159 5600 raft_consensus.cc:357] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.451845 5600 raft_consensus.cc:383] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.451692 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_26e52fb26ebe44ef9a2b6e2d51defe22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:03.452119 5600 raft_consensus.cc:738] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.454289 5593 ts_tablet_manager.cc:1939] T 80a22306f7c7487d996969e349f09766 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:03.454972 5595 ts_tablet_manager.cc:1918] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.455492 5595 log.cc:1198] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/80a22306f7c7487d996969e349f09766
I20250110 20:57:03.456377 5595 ts_tablet_manager.cc:1939] T 80a22306f7c7487d996969e349f09766 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:03.457696 5600 consensus_queue.cc:260] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.459619 5593 tablet_replica.cc:331] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:03.460348 5593 raft_consensus.cc:2238] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.460786 5593 raft_consensus.cc:2267] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.468604 5593 ts_tablet_manager.cc:1905] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.478153 5600 ts_tablet_manager.cc:1428] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.029s user 0.006s sys 0.002s
I20250110 20:57:03.490353 5600 tablet_bootstrap.cc:492] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:03.471261 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 80a22306f7c7487d996969e349f09766 (table default.Table_e8fd956943c647e1890ffe3f3d3597f7 [id=3fb5402eb3a944d989a916f70a71c2d3]) successfully deleted
W20250110 20:57:03.499514 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26e52fb26ebe44ef9a2b6e2d51defe22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:03.506498 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_213a26b346b94108a30a80c719397f12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.506536 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb with delete_type TABLET_DATA_DELETED (replaced by tablet 01962c5b7ae64f80b998948d18250b4c) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:03.508379 5595 tablet_replica.cc:331] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:03.509069 5595 raft_consensus.cc:2238] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.509519 5595 raft_consensus.cc:2267] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.512233 5595 ts_tablet_manager.cc:1905] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.513108 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 80a22306f7c7487d996969e349f09766 (table default.Table_e8fd956943c647e1890ffe3f3d3597f7 [id=3fb5402eb3a944d989a916f70a71c2d3]) successfully deleted
I20250110 20:57:03.521623 5594 tablet_bootstrap.cc:492] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:03.522045 5594 ts_tablet_manager.cc:1397] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.250s user 0.000s sys 0.014s
I20250110 20:57:03.530054 5600 tablet_bootstrap.cc:654] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:03.540714 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 has no consensus state
I20250110 20:57:03.552713 4891 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_0fb62c9c09f741c7beea780971c66f3a" } modify_external_catalogs: true
I20250110 20:57:03.553243 4891 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_0fb62c9c09f741c7beea780971c66f3a" } modify_external_catalogs: true
I20250110 20:57:03.555126 4895 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_6c07fcebfdf6402a8876074a13c5708b" } modify_external_catalogs: true
I20250110 20:57:03.555622 4895 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_6c07fcebfdf6402a8876074a13c5708b" } modify_external_catalogs: true
I20250110 20:57:03.557631 5593 ts_tablet_manager.cc:1918] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.563349 5593 log.cc:1198] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ac9c44d7b9364ccdbf7ca4c3bcf692fb
I20250110 20:57:03.565801 4898 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2fee2488f5df4696853086b480b0a2e8" } new_table_name: "default.Table_d2f92229bca74d26baf2508b363c90ad" modify_external_catalogs: true
I20250110 20:57:03.549870 5594 raft_consensus.cc:357] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.570274 5594 raft_consensus.cc:383] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.570573 5594 raft_consensus.cc:738] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.572144 5595 ts_tablet_manager.cc:1918] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.572438 5595 log.cc:1198] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ac9c44d7b9364ccdbf7ca4c3bcf692fb
I20250110 20:57:03.573071 5595 ts_tablet_manager.cc:1939] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:03.572789 5594 consensus_queue.cc:260] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.574983 5592 ts_tablet_manager.cc:1918] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.575387 5592 log.cc:1198] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/5c2e7b5f8e6440ba8fb8890132781449
I20250110 20:57:03.576187 5592 ts_tablet_manager.cc:1939] T 5c2e7b5f8e6440ba8fb8890132781449 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:03.579324 5594 ts_tablet_manager.cc:1428] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.057s user 0.006s sys 0.000s
I20250110 20:57:03.579596 5600 tablet_bootstrap.cc:492] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:03.579985 5600 ts_tablet_manager.cc:1397] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.003s
I20250110 20:57:03.580156 5594 tablet_bootstrap.cc:492] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:03.574429 5597 tablet_bootstrap.cc:492] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.582293 5597 ts_tablet_manager.cc:1397] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.142s user 0.035s sys 0.038s
I20250110 20:57:03.582834 5593 ts_tablet_manager.cc:1939] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:03.583834 5600 raft_consensus.cc:357] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.584489 5600 raft_consensus.cc:383] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.584746 5600 raft_consensus.cc:738] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.584213 4900 catalog_manager.cc:5526] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:03.585299 5600 consensus_queue.cc:260] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.584408 5597 raft_consensus.cc:357] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:03.591104 5597 raft_consensus.cc:383] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.591367 5597 raft_consensus.cc:738] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.591924 5597 consensus_queue.cc:260] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:03.592291 4892 catalog_manager.cc:5526] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:03.594583 5597 ts_tablet_manager.cc:1428] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.012s user 0.001s sys 0.004s
I20250110 20:57:03.594640 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (DEFAULT_TABLE table=default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.595396 5597 tablet_bootstrap.cc:492] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.595616 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.596777 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (DEFAULT_TABLE table=default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.597968 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.604676 5302 tablet_service.cc:1467] Processing CreateTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 (DEFAULT_TABLE table=default.Table_ccaf097e265d4ed6a3ed2e9c0984229a [id=af9522c35cd74c119bdc1af28f7bab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.605726 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bc3aae47f54c618798ff30c9cb75e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.607543 5442 tablet_service.cc:1467] Processing CreateTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 (DEFAULT_TABLE table=default.Table_ccaf097e265d4ed6a3ed2e9c0984229a [id=af9522c35cd74c119bdc1af28f7bab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.608501 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bc3aae47f54c618798ff30c9cb75e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.614010 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (DEFAULT_TABLE table=default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.614987 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.618829 5597 tablet_bootstrap.cc:654] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:03.618796 4885 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 consensus state has no leader
I20250110 20:57:03.619407 5154 tablet_service.cc:1467] Processing CreateTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 (DEFAULT_TABLE table=default.Table_ccaf097e265d4ed6a3ed2e9c0984229a [id=af9522c35cd74c119bdc1af28f7bab6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.620347 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7bc3aae47f54c618798ff30c9cb75e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.619933 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_b5563b54e22d43e88062859e6da9bc4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:03.622335 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c2e7b5f8e6440ba8fb8890132781449 with delete_type TABLET_DATA_DELETED (replaced by tablet 637dda10276947d5842ed24bbf044d0d) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:03.623119 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5563b54e22d43e88062859e6da9bc4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.623332 5600 ts_tablet_manager.cc:1428] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20250110 20:57:03.623410 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb (table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]) successfully deleted
I20250110 20:57:03.624598 5600 tablet_bootstrap.cc:492] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:03.634265 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb (table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]) successfully deleted
I20250110 20:57:03.635483 5595 tablet_replica.cc:331] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:03.636211 5595 raft_consensus.cc:2238] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.636682 5595 raft_consensus.cc:2267] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.640923 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 01962c5b7ae64f80b998948d18250b4c (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.641553 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 01962c5b7ae64f80b998948d18250b4c (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.641934 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01962c5b7ae64f80b998948d18250b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.642539 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01962c5b7ae64f80b998948d18250b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.647081 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 01962c5b7ae64f80b998948d18250b4c (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.648625 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01962c5b7ae64f80b998948d18250b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.666879 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 5c2e7b5f8e6440ba8fb8890132781449 (table default.Table_54d6bd0b8362483ba4440d3d954c1383 [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
W20250110 20:57:03.676206 4898 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dc0d3abef5741339fdbb9abeed20b15 consensus state has no leader
I20250110 20:57:03.681912 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 332dfe3a0e45468695aeb7d7eb7c30ca
I20250110 20:57:03.683220 4897 catalog_manager.cc:6596] ReplaceTablet: tablet 332dfe3a0e45468695aeb7d7eb7c30ca deleted and replaced by tablet a7e7ffb063594128ba137f9aea86bf15
I20250110 20:57:03.683590 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca with delete_type TABLET_DATA_DELETED (replaced by tablet a7e7ffb063594128ba137f9aea86bf15) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:03.684286 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca with delete_type TABLET_DATA_DELETED (replaced by tablet a7e7ffb063594128ba137f9aea86bf15) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:03.685014 5593 tablet_replica.cc:331] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:03.685106 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 7680441899e14a2d9a6ff45d0e0d69b8
I20250110 20:57:03.686220 5593 raft_consensus.cc:2238] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.686707 5593 raft_consensus.cc:2267] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.687465 5595 ts_tablet_manager.cc:1905] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.689195 5593 ts_tablet_manager.cc:1905] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.689175 5624 raft_consensus.cc:491] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.689601 5624 raft_consensus.cc:513] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.691342 5624 leader_election.cc:290] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:03.693087 5398 leader_election.cc:343] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: ac9c44d7b9364ccdbf7ca4c3bcf692fb
W20250110 20:57:03.693861 5399 leader_election.cc:343] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: ac9c44d7b9364ccdbf7ca4c3bcf692fb
I20250110 20:57:03.694240 5399 leader_election.cc:304] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:03.696723 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca with delete_type TABLET_DATA_DELETED (replaced by tablet a7e7ffb063594128ba137f9aea86bf15) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:03.699714 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:03.700719 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:03.701193 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:03.703624 5624 raft_consensus.cc:2743] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:03.706175 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 332dfe3a0e45468695aeb7d7eb7c30ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 332dfe3a0e45468695aeb7d7eb7c30ca already in progress: creating tablet
W20250110 20:57:03.710750 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 7680441899e14a2d9a6ff45d0e0d69b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7680441899e14a2d9a6ff45d0e0d69b8 already in progress: creating tablet
I20250110 20:57:03.710906 5597 tablet_bootstrap.cc:492] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.711261 5597 ts_tablet_manager.cc:1397] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.116s user 0.025s sys 0.003s
I20250110 20:57:03.713294 5597 raft_consensus.cc:357] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.714756 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ec98243c2d784adfb3ab2a95bd2d89b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:03.715296 5597 raft_consensus.cc:383] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.715533 5597 raft_consensus.cc:738] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
W20250110 20:57:03.716271 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec98243c2d784adfb3ab2a95bd2d89b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.716162 5597 consensus_queue.cc:260] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.720533 5597 ts_tablet_manager.cc:1428] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20250110 20:57:03.728767 5593 ts_tablet_manager.cc:1918] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.722386 5600 tablet_bootstrap.cc:654] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.721302 5597 tablet_bootstrap.cc:492] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.738366 5594 tablet_bootstrap.cc:654] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.741448 5597 tablet_bootstrap.cc:654] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.744102 5593 log.cc:1198] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/332dfe3a0e45468695aeb7d7eb7c30ca
I20250110 20:57:03.744714 5593 ts_tablet_manager.cc:1939] T 332dfe3a0e45468695aeb7d7eb7c30ca P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:03.745254 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb with delete_type TABLET_DATA_DELETED (replaced by tablet 01962c5b7ae64f80b998948d18250b4c) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:03.745941 5592 tablet_replica.cc:331] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:03.746678 5592 raft_consensus.cc:2238] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.747182 5592 raft_consensus.cc:2267] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.747438 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 332dfe3a0e45468695aeb7d7eb7c30ca (table default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]) successfully deleted
I20250110 20:57:03.748744 5593 tablet_replica.cc:331] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:03.749280 5592 ts_tablet_manager.cc:1905] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.749490 5593 raft_consensus.cc:2238] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.749501 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_653008e13c0c4667a68753a92ae1a596"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:03.749931 5593 raft_consensus.cc:2267] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:03.751130 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_653008e13c0c4667a68753a92ae1a596 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.751834 5593 ts_tablet_manager.cc:1905] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.753957 4892 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_54d6bd0b8362483ba4440d3d954c1383" } new_table_name: "default.Table_d5cee3d339fd4e289d63ec604aface2e" modify_external_catalogs: true
I20250110 20:57:03.757164 5597 tablet_bootstrap.cc:492] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.757544 5597 ts_tablet_manager.cc:1397] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.000s
I20250110 20:57:03.759567 5597 raft_consensus.cc:357] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:03.760174 5597 raft_consensus.cc:383] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.760409 5597 raft_consensus.cc:738] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.760944 5597 consensus_queue.cc:260] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:03.763530 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_64d7746346a94643bdf6debc10a1ac6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:03.765017 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64d7746346a94643bdf6debc10a1ac6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.767123 5600 tablet_bootstrap.cc:492] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:03.767508 5600 ts_tablet_manager.cc:1397] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.143s user 0.005s sys 0.005s
I20250110 20:57:03.770202 5600 raft_consensus.cc:357] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.770882 5600 raft_consensus.cc:383] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.771157 5600 raft_consensus.cc:738] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.771733 5600 consensus_queue.cc:260] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.777127 5597 ts_tablet_manager.cc:1428] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250110 20:57:03.777377 5594 tablet_bootstrap.cc:492] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:03.778142 5594 ts_tablet_manager.cc:1397] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.198s user 0.005s sys 0.008s
I20250110 20:57:03.781673 5594 raft_consensus.cc:357] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.782047 5595 ts_tablet_manager.cc:1918] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.782398 5595 log.cc:1198] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/5c2e7b5f8e6440ba8fb8890132781449
I20250110 20:57:03.784446 5597 tablet_bootstrap.cc:492] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.784724 5594 raft_consensus.cc:383] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.785041 5594 raft_consensus.cc:738] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.787456 5595 ts_tablet_manager.cc:1939] T 5c2e7b5f8e6440ba8fb8890132781449 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:03.792382 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 5c2e7b5f8e6440ba8fb8890132781449 (table default.Table_54d6bd0b8362483ba4440d3d954c1383 [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
W20250110 20:57:03.795481 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 332dfe3a0e45468695aeb7d7eb7c30ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 332dfe3a0e45468695aeb7d7eb7c30ca already in progress: creating tablet
I20250110 20:57:03.798671 5592 ts_tablet_manager.cc:1918] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 20:57:03.799212 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 7680441899e14a2d9a6ff45d0e0d69b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7680441899e14a2d9a6ff45d0e0d69b8 already in progress: creating tablet
I20250110 20:57:03.802873 5597 tablet_bootstrap.cc:654] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.803228 5592 log.cc:1198] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ac9c44d7b9364ccdbf7ca4c3bcf692fb
I20250110 20:57:03.803951 5592 ts_tablet_manager.cc:1939] T ac9c44d7b9364ccdbf7ca4c3bcf692fb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:03.789381 5594 consensus_queue.cc:260] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.804220 5593 ts_tablet_manager.cc:1918] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.808198 5600 ts_tablet_manager.cc:1428] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.040s user 0.007s sys 0.001s
I20250110 20:57:03.809473 5593 log.cc:1198] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/7680441899e14a2d9a6ff45d0e0d69b8
I20250110 20:57:03.810286 5593 ts_tablet_manager.cc:1939] T 7680441899e14a2d9a6ff45d0e0d69b8 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:03.811242 5600 tablet_bootstrap.cc:492] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:03.814843 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 7680441899e14a2d9a6ff45d0e0d69b8 (table default.Table_6c07fcebfdf6402a8876074a13c5708b [id=7a14b963398847d6b19313baae9772ea]) successfully deleted
I20250110 20:57:03.816027 5594 ts_tablet_manager.cc:1428] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.038s user 0.000s sys 0.006s
I20250110 20:57:03.818499 5594 tablet_bootstrap.cc:492] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:03.823032 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet ac9c44d7b9364ccdbf7ca4c3bcf692fb (table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]) successfully deleted
I20250110 20:57:03.826473 5600 tablet_bootstrap.cc:654] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.830377 5594 tablet_bootstrap.cc:654] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.835249 5581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:38225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:34617,127.4.171.253:34381,127.4.171.252:37233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.252:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.171.252
--webserver_port=45215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:03.838330 5597 tablet_bootstrap.cc:492] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.838749 5597 ts_tablet_manager.cc:1397] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.055s user 0.008s sys 0.001s
I20250110 20:57:03.840767 5597 raft_consensus.cc:357] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.841090 5581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:03.841368 5597 raft_consensus.cc:383] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.841629 5597 raft_consensus.cc:738] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.842272 5597 consensus_queue.cc:260] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.844219 5581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250110 20:57:03.845137 5624 raft_consensus.cc:491] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.845551 5624 raft_consensus.cc:513] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.847275 5624 leader_election.cc:290] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:03.847507 5597 ts_tablet_manager.cc:1428] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250110 20:57:03.848256 5597 tablet_bootstrap.cc:492] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:03.848305 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd2cc66ab594f849f142d7847a1e0f5" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:03.848773 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fd2cc66ab594f849f142d7847a1e0f5" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:03.849064 5323 raft_consensus.cc:2388] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in current term 1: Already voted for candidate ac9003be04f340f3886b5625cc928eb0 in this term.
I20250110 20:57:03.850638 5399 leader_election.cc:304] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:03.851653 5624 raft_consensus.cc:3054] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:03.853097 5600 tablet_bootstrap.cc:492] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:03.853483 5600 ts_tablet_manager.cc:1397] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.000s
I20250110 20:57:03.856002 5597 tablet_bootstrap.cc:654] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.856087 5600 raft_consensus.cc:357] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:03.856694 5600 raft_consensus.cc:383] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.856943 5600 raft_consensus.cc:738] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.857509 5600 consensus_queue.cc:260] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:03.859360 5606 raft_consensus.cc:491] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.859773 5606 raft_consensus.cc:513] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.859915 5624 raft_consensus.cc:2743] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:03.861585 5606 leader_election.cc:290] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:03.864334 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bc3aae47f54c618798ff30c9cb75e8" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:03.866214 5110 leader_election.cc:343] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.866520 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bc3aae47f54c618798ff30c9cb75e8" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:03.867460 5600 ts_tablet_manager.cc:1428] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20250110 20:57:03.868899 5612 raft_consensus.cc:491] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.869314 5612 raft_consensus.cc:513] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:03.870043 5111 leader_election.cc:343] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.870445 5111 leader_election.cc:304] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:03.874011 5606 raft_consensus.cc:2743] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:03.875875 5594 tablet_bootstrap.cc:492] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:03.876329 5594 ts_tablet_manager.cc:1397] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.004s
I20250110 20:57:03.879554 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b5563b54e22d43e88062859e6da9bc4a" } new_table_name: "default.Table_1367656b1bef487abdee0e294f835b5f" modify_external_catalogs: true
I20250110 20:57:03.880192 5612 leader_election.cc:290] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:03.886512 5594 raft_consensus.cc:357] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.887257 5594 raft_consensus.cc:383] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.887565 5594 raft_consensus.cc:738] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.888221 5594 consensus_queue.cc:260] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.896919 5594 ts_tablet_manager.cc:1428] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20250110 20:57:03.897810 5594 tablet_bootstrap.cc:492] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:03.868639 5600 tablet_bootstrap.cc:492] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:03.905999 4898 master_service.cc:946] ReplaceTablet: received request to replace tablet a04595ff7d1e4402a486f4ddf420ea29 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:03.907281 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ccaf097e265d4ed6a3ed2e9c0984229a" } modify_external_catalogs: true
I20250110 20:57:03.907790 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ccaf097e265d4ed6a3ed2e9c0984229a" } modify_external_catalogs: true
I20250110 20:57:03.907977 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_13547221ce544ee7bf2f209caac6d5d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:03.909818 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13547221ce544ee7bf2f209caac6d5d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:03.917268 4893 master_service.cc:946] ReplaceTablet: received request to replace tablet a04595ff7d1e4402a486f4ddf420ea29 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:03.930310 5594 tablet_bootstrap.cc:654] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.940924 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 574171c94ca6450daf9fb608dbe9b002 (DEFAULT_TABLE table=default.Table_26e52fb26ebe44ef9a2b6e2d51defe22 [id=44b872c361cf41a0b7ec0d28e66725ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.941387 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:03.941951 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574171c94ca6450daf9fb608dbe9b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.942207 5595 tablet_replica.cc:331] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:03.942167 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 574171c94ca6450daf9fb608dbe9b002 (DEFAULT_TABLE table=default.Table_26e52fb26ebe44ef9a2b6e2d51defe22 [id=44b872c361cf41a0b7ec0d28e66725ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.942963 5595 raft_consensus.cc:2238] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.943212 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574171c94ca6450daf9fb608dbe9b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.943403 5595 raft_consensus.cc:2267] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.945770 5595 ts_tablet_manager.cc:1905] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:03.952137 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5165408eb95e4849886252671dfb18d4 has no consensus state
I20250110 20:57:03.958341 5600 tablet_bootstrap.cc:654] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:03.964913 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 574171c94ca6450daf9fb608dbe9b002 (DEFAULT_TABLE table=default.Table_26e52fb26ebe44ef9a2b6e2d51defe22 [id=44b872c361cf41a0b7ec0d28e66725ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.966056 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 574171c94ca6450daf9fb608dbe9b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.971553 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332dfe3a0e45468695aeb7d7eb7c30ca" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:03.973671 5255 leader_election.cc:343] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:03.969023 5595 ts_tablet_manager.cc:1918] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:03.975207 5595 log.cc:1198] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/7680441899e14a2d9a6ff45d0e0d69b8
I20250110 20:57:03.975910 5595 ts_tablet_manager.cc:1939] T 7680441899e14a2d9a6ff45d0e0d69b8 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:03.978101 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet e7bc3aae47f54c618798ff30c9cb75e8
I20250110 20:57:03.978269 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 7680441899e14a2d9a6ff45d0e0d69b8 (table default.Table_6c07fcebfdf6402a8876074a13c5708b [id=7a14b963398847d6b19313baae9772ea]) successfully deleted
I20250110 20:57:03.980132 5597 tablet_bootstrap.cc:492] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:03.980504 5597 ts_tablet_manager.cc:1397] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.132s user 0.038s sys 0.039s
W20250110 20:57:03.980721 5257 leader_election.cc:343] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: 332dfe3a0e45468695aeb7d7eb7c30ca
I20250110 20:57:03.981086 5257 leader_election.cc:304] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:03.981709 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:03.982378 5612 raft_consensus.cc:2743] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:03.982889 5624 raft_consensus.cc:491] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:03.982641 5597 raft_consensus.cc:357] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.983248 5597 raft_consensus.cc:383] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:03.983493 5597 raft_consensus.cc:738] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:03.983276 5624 raft_consensus.cc:513] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.985004 5624 leader_election.cc:290] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:03.985596 5600 tablet_bootstrap.cc:492] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:03.986001 5600 ts_tablet_manager.cc:1397] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.000s
W20250110 20:57:03.987193 5398 leader_election.cc:343] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 7680441899e14a2d9a6ff45d0e0d69b8
W20250110 20:57:03.987269 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet e7bc3aae47f54c618798ff30c9cb75e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7bc3aae47f54c618798ff30c9cb75e8 already in progress: creating tablet
W20250110 20:57:03.988040 5399 leader_election.cc:343] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: 7680441899e14a2d9a6ff45d0e0d69b8
I20250110 20:57:03.988406 5399 leader_election.cc:304] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:03.989125 5624 raft_consensus.cc:2743] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:03.989897 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:03.990121 5649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:03.990603 5593 tablet_replica.cc:331] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:03.991310 5593 raft_consensus.cc:2238] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:03.991729 5593 raft_consensus.cc:2267] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:03.987826 5597 consensus_queue.cc:260] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:03.993142 5444 tablet_service.cc:1467] Processing CreateTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a (DEFAULT_TABLE table=default.Table_aacabce3495d4b85b5f97043bc6a7242 [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.993963 5593 ts_tablet_manager.cc:1905] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:03.994168 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26f58b066f646e5b11ddf20cbbf9e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.994750 5153 tablet_service.cc:1467] Processing CreateTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a (DEFAULT_TABLE table=default.Table_aacabce3495d4b85b5f97043bc6a7242 [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.995710 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26f58b066f646e5b11ddf20cbbf9e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.997483 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 (DEFAULT_TABLE table=default.Table_cd3dd0abc91e4209b6f8586377dafc01 [id=3bbd1acd709b44899907148df944b20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:03.998932 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29599cfce9ba4875b7aed08f5094ce21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:03.999626 5302 tablet_service.cc:1467] Processing CreateTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a (DEFAULT_TABLE table=default.Table_aacabce3495d4b85b5f97043bc6a7242 [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.000607 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b26f58b066f646e5b11ddf20cbbf9e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.004545 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.007445 5600 raft_consensus.cc:357] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.008085 5600 raft_consensus.cc:383] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:04.008625 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet e7bc3aae47f54c618798ff30c9cb75e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7bc3aae47f54c618798ff30c9cb75e8 already in progress: creating tablet
I20250110 20:57:04.008888 5600 raft_consensus.cc:738] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.009469 5600 consensus_queue.cc:260] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.010685 5597 ts_tablet_manager.cc:1428] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20250110 20:57:04.010959 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_18d27dbc0aca402787c14b0044972cb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:04.012344 5597 tablet_bootstrap.cc:492] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:04.012511 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18d27dbc0aca402787c14b0044972cb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.016870 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 (DEFAULT_TABLE table=default.Table_cd3dd0abc91e4209b6f8586377dafc01 [id=3bbd1acd709b44899907148df944b20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.018335 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29599cfce9ba4875b7aed08f5094ce21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.018615 5600 ts_tablet_manager.cc:1428] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250110 20:57:04.020284 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 (DEFAULT_TABLE table=default.Table_cd3dd0abc91e4209b6f8586377dafc01 [id=3bbd1acd709b44899907148df944b20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.021241 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29599cfce9ba4875b7aed08f5094ce21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.023869 5600 tablet_bootstrap.cc:492] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.029104 5600 tablet_bootstrap.cc:654] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.036003 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 22b5876f954e469f86498a529e5c4a68 (DEFAULT_TABLE table=default.Table_213a26b346b94108a30a80c719397f12 [id=3294cc03112a4958b6a090385afd9790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.037043 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b5876f954e469f86498a529e5c4a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.044313 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_3023e8331a774a0e911891f54fe766c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:04.045866 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3023e8331a774a0e911891f54fe766c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.061851 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 22b5876f954e469f86498a529e5c4a68 (DEFAULT_TABLE table=default.Table_213a26b346b94108a30a80c719397f12 [id=3294cc03112a4958b6a090385afd9790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.062994 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b5876f954e469f86498a529e5c4a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.063829 4898 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet a04595ff7d1e4402a486f4ddf420ea29
I20250110 20:57:04.065177 4898 catalog_manager.cc:6596] ReplaceTablet: tablet a04595ff7d1e4402a486f4ddf420ea29 deleted and replaced by tablet 5256d081843a492d8081c4319f3446d5
I20250110 20:57:04.067529 4900 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_54d6bd0b8362483ba4440d3d954c1383 [id=896c7b90b1bd4d048082ccac12d70301]). Expected version 2 got 0
I20250110 20:57:04.067814 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet a04595ff7d1e4402a486f4ddf420ea29 with delete_type TABLET_DATA_DELETED (replaced by tablet 5256d081843a492d8081c4319f3446d5) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:04.068713 5592 tablet_replica.cc:331] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:04.069448 5592 raft_consensus.cc:2238] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:04.070075 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet a04595ff7d1e4402a486f4ddf420ea29 with delete_type TABLET_DATA_DELETED (replaced by tablet 5256d081843a492d8081c4319f3446d5) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.070215 5592 pending_rounds.cc:62] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:04.070472 5592 pending_rounds.cc:69] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878585386467328 op_type: NO_OP noop_request { }
I20250110 20:57:04.070739 5595 tablet_replica.cc:331] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:04.070940 5592 raft_consensus.cc:2883] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20250110 20:57:04.071249 4892 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 637dda10276947d5842ed24bbf044d0d consensus state has no leader
I20250110 20:57:04.071265 5592 raft_consensus.cc:2267] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.071400 5595 raft_consensus.cc:2238] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:04.071812 5595 raft_consensus.cc:2267] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.073534 4891 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet a7e7ffb063594128ba137f9aea86bf15
I20250110 20:57:04.074029 5595 ts_tablet_manager.cc:1905] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.082027 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.082455 5592 ts_tablet_manager.cc:1905] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.082518 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:04.083235 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.103818 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_1d485a05ae354e4492c99bf9c67d6913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:04.105862 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d485a05ae354e4492c99bf9c67d6913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.111541 5595 ts_tablet_manager.cc:1918] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:04.104044 5592 ts_tablet_manager.cc:1918] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:04.112843 5592 log.cc:1198] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/a04595ff7d1e4402a486f4ddf420ea29
I20250110 20:57:04.113554 5592 ts_tablet_manager.cc:1939] T a04595ff7d1e4402a486f4ddf420ea29 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:04.116050 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet a04595ff7d1e4402a486f4ddf420ea29 (table default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]) successfully deleted
I20250110 20:57:04.033372 5597 tablet_bootstrap.cc:654] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.040122 5152 tablet_service.cc:1467] Processing CreateTablet for tablet 22b5876f954e469f86498a529e5c4a68 (DEFAULT_TABLE table=default.Table_213a26b346b94108a30a80c719397f12 [id=3294cc03112a4958b6a090385afd9790]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.117794 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22b5876f954e469f86498a529e5c4a68. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.119544 5594 tablet_bootstrap.cc:492] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:04.119948 5594 ts_tablet_manager.cc:1397] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.222s user 0.010s sys 0.009s
I20250110 20:57:04.078450 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 7680441899e14a2d9a6ff45d0e0d69b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:04.078683 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet a04595ff7d1e4402a486f4ddf420ea29 with delete_type TABLET_DATA_DELETED (replaced by tablet 5256d081843a492d8081c4319f3446d5) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:04.088248 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:04.021255 5593 ts_tablet_manager.cc:1918] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:04.122750 5594 raft_consensus.cc:357] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.123397 5593 log.cc:1198] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/e7bc3aae47f54c618798ff30c9cb75e8
I20250110 20:57:04.123417 5594 raft_consensus.cc:383] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.123677 5594 raft_consensus.cc:738] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.124030 5593 ts_tablet_manager.cc:1939] T e7bc3aae47f54c618798ff30c9cb75e8 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:04.124259 5594 consensus_queue.cc:260] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.126770 5593 tablet_replica.cc:331] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:04.130381 5593 raft_consensus.cc:2238] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:04.131117 5593 raft_consensus.cc:2267] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.133191 5594 ts_tablet_manager.cc:1428] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20250110 20:57:04.136091 5593 ts_tablet_manager.cc:1905] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.136094 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet e7bc3aae47f54c618798ff30c9cb75e8 (table default.Table_ccaf097e265d4ed6a3ed2e9c0984229a [id=af9522c35cd74c119bdc1af28f7bab6c]) successfully deleted
W20250110 20:57:04.136718 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 7680441899e14a2d9a6ff45d0e0d69b8 because the tablet was not found. No further retry: Not found: Tablet not found: 7680441899e14a2d9a6ff45d0e0d69b8
I20250110 20:57:04.138597 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 80a22306f7c7487d996969e349f09766 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:02 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:04.145653 5597 tablet_bootstrap.cc:492] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:04.146072 5597 ts_tablet_manager.cc:1397] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.134s user 0.007s sys 0.002s
I20250110 20:57:04.147236 5594 tablet_bootstrap.cc:492] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:04.148303 5597 raft_consensus.cc:357] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.153581 5597 raft_consensus.cc:383] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.153832 5597 raft_consensus.cc:738] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.155285 5594 tablet_bootstrap.cc:654] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.155457 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_aacabce3495d4b85b5f97043bc6a7242" } new_table_name: "default.Table_c97acba407484513a151d33ba86bd628" modify_external_catalogs: true
I20250110 20:57:04.161335 5606 raft_consensus.cc:491] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:04.161787 5606 raft_consensus.cc:513] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.162285 5592 tablet_replica.cc:331] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:04.162971 5592 raft_consensus.cc:2238] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:04.163404 5592 raft_consensus.cc:2267] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.163559 5606 leader_election.cc:290] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:04.165041 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:04.166778 5110 leader_election.cc:343] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.166014 5595 log.cc:1198] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/a04595ff7d1e4402a486f4ddf420ea29
I20250110 20:57:04.167886 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:04.167886 5595 ts_tablet_manager.cc:1939] T a04595ff7d1e4402a486f4ddf420ea29 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
W20250110 20:57:04.169277 5111 leader_election.cc:343] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.169661 5111 leader_election.cc:304] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:04.171166 5606 raft_consensus.cc:2743] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:04.171346 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet a04595ff7d1e4402a486f4ddf420ea29 (table default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]) successfully deleted
W20250110 20:57:04.172793 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 7680441899e14a2d9a6ff45d0e0d69b8 because the tablet was not found. No further retry: Not found: Tablet not found: 7680441899e14a2d9a6ff45d0e0d69b8
I20250110 20:57:04.173447 5600 tablet_bootstrap.cc:492] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.173836 5600 ts_tablet_manager.cc:1397] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.150s user 0.022s sys 0.031s
I20250110 20:57:04.174126 5592 ts_tablet_manager.cc:1905] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.174247 5595 tablet_replica.cc:331] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:04.175961 5600 raft_consensus.cc:357] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.176604 5600 raft_consensus.cc:383] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.176843 5600 raft_consensus.cc:738] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.154217 5597 consensus_queue.cc:260] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.180681 5600 consensus_queue.cc:260] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:04.181232 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b26f58b066f646e5b11ddf20cbbf9e1a consensus state has no leader
I20250110 20:57:04.183388 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e570881031794b9ea4fd25d7b6fd4198"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:04.185534 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e570881031794b9ea4fd25d7b6fd4198 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.188186 4894 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1367656b1bef487abdee0e294f835b5f" } modify_external_catalogs: true
I20250110 20:57:04.188726 4894 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1367656b1bef487abdee0e294f835b5f" } modify_external_catalogs: true
I20250110 20:57:04.189678 5600 ts_tablet_manager.cc:1428] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250110 20:57:04.192212 5597 ts_tablet_manager.cc:1428] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.046s user 0.005s sys 0.000s
I20250110 20:57:04.176332 5595 raft_consensus.cc:2238] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:04.192584 5593 ts_tablet_manager.cc:1918] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:04.192874 5595 raft_consensus.cc:2267] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.192952 5593 log.cc:1198] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/a04595ff7d1e4402a486f4ddf420ea29
I20250110 20:57:04.194448 5600 tablet_bootstrap.cc:492] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.196295 5594 tablet_bootstrap.cc:492] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:04.196668 5594 ts_tablet_manager.cc:1397] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.009s
I20250110 20:57:04.199165 5593 ts_tablet_manager.cc:1939] T a04595ff7d1e4402a486f4ddf420ea29 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:04.200223 5595 ts_tablet_manager.cc:1905] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.201562 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet a04595ff7d1e4402a486f4ddf420ea29 (table default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]) successfully deleted
W20250110 20:57:04.202258 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 332dfe3a0e45468695aeb7d7eb7c30ca because the tablet was not found. No further retry: Not found: Tablet not found: 332dfe3a0e45468695aeb7d7eb7c30ca
I20250110 20:57:04.208513 5594 raft_consensus.cc:357] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.209146 5594 raft_consensus.cc:383] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.209375 5594 raft_consensus.cc:738] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.215073 5597 tablet_bootstrap.cc:492] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:04.217700 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]). Expected version 3 got 0
I20250110 20:57:04.219074 5592 ts_tablet_manager.cc:1918] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:04.219445 5592 log.cc:1198] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/7680441899e14a2d9a6ff45d0e0d69b8
I20250110 20:57:04.221077 4900 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 2dc0d3abef5741339fdbb9abeed20b15 (table default.Table_d2f92229bca74d26baf2508b363c90ad [id=9daaee4eb9574713b7914a551680c52b]). Expected version 2 got 0
I20250110 20:57:04.222105 5592 ts_tablet_manager.cc:1939] T 7680441899e14a2d9a6ff45d0e0d69b8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:04.226073 5441 tablet_service.cc:1467] Processing CreateTablet for tablet e38b659f218d42298cb36752b5ca603d (DEFAULT_TABLE table=default.Table_13547221ce544ee7bf2f209caac6d5d8 [id=ba089f4f57284f80ac3621a88610fec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.227054 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38b659f218d42298cb36752b5ca603d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.225075 5300 tablet_service.cc:1467] Processing CreateTablet for tablet e38b659f218d42298cb36752b5ca603d (DEFAULT_TABLE table=default.Table_13547221ce544ee7bf2f209caac6d5d8 [id=ba089f4f57284f80ac3621a88610fec1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.230082 4899 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2dc0d3abef5741339fdbb9abeed20b15 (table default.Table_d2f92229bca74d26baf2508b363c90ad [id=9daaee4eb9574713b7914a551680c52b]). Expected version 2 got 0
I20250110 20:57:04.231158 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38b659f218d42298cb36752b5ca603d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.231552 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 472174c6885b446a839721157914f047 (DEFAULT_TABLE table=default.Table_64d7746346a94643bdf6debc10a1ac6e [id=95036a214c1740d4b8391e06f7110688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.232525 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472174c6885b446a839721157914f047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.234486 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 5165408eb95e4849886252671dfb18d4 (DEFAULT_TABLE table=default.Table_1367656b1bef487abdee0e294f835b5f [id=ef270c5c9b0d465b8333565e19edd86b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.235452 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5165408eb95e4849886252671dfb18d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.227653 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 472174c6885b446a839721157914f047 (DEFAULT_TABLE table=default.Table_64d7746346a94643bdf6debc10a1ac6e [id=95036a214c1740d4b8391e06f7110688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.238155 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472174c6885b446a839721157914f047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.239056 5299 tablet_service.cc:1467] Processing CreateTablet for tablet a7e7ffb063594128ba137f9aea86bf15 (DEFAULT_TABLE table=default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.240453 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 9155a626d26541ce9d0cf31508f19f15 (DEFAULT_TABLE table=default.Table_ec98243c2d784adfb3ab2a95bd2d89b8 [id=62486535b69d4a4798776d26dfdce501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.240995 5298 tablet_service.cc:1467] Processing CreateTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 (DEFAULT_TABLE table=default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.241675 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9155a626d26541ce9d0cf31508f19f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.242333 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e7ffb063594128ba137f9aea86bf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.232982 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 9155a626d26541ce9d0cf31508f19f15 (DEFAULT_TABLE table=default.Table_ec98243c2d784adfb3ab2a95bd2d89b8 [id=62486535b69d4a4798776d26dfdce501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.243546 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9155a626d26541ce9d0cf31508f19f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.245647 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ac7d62f978433dad004f2908c4bbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.246285 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 5165408eb95e4849886252671dfb18d4 (DEFAULT_TABLE table=default.Table_1367656b1bef487abdee0e294f835b5f [id=ef270c5c9b0d465b8333565e19edd86b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.247323 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5165408eb95e4849886252671dfb18d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.250780 5440 tablet_service.cc:1467] Processing CreateTablet for tablet a7e7ffb063594128ba137f9aea86bf15 (DEFAULT_TABLE table=default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.251778 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e7ffb063594128ba137f9aea86bf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.254293 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 (DEFAULT_TABLE table=default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.255370 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ac7d62f978433dad004f2908c4bbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.257859 5594 consensus_queue.cc:260] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.262305 5600 tablet_bootstrap.cc:654] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.278983 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_13547221ce544ee7bf2f209caac6d5d8" } new_table_name: "default.Table_635ff4f15ebf4d6580736411eaa6b821" modify_external_catalogs: true
I20250110 20:57:04.279877 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_14adb01640aa40d699cfb94980d7aea3" } modify_external_catalogs: true
I20250110 20:57:04.281793 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_14adb01640aa40d699cfb94980d7aea3" } modify_external_catalogs: true
I20250110 20:57:04.289341 4894 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 5165408eb95e4849886252671dfb18d4
I20250110 20:57:04.291034 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet 5165408eb95e4849886252671dfb18d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.295785 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 332dfe3a0e45468695aeb7d7eb7c30ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:04.301004 5437 tablet_service.cc:1514] Processing DeleteTablet for tablet 5165408eb95e4849886252671dfb18d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:04.302197 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 7680441899e14a2d9a6ff45d0e0d69b8 (table default.Table_6c07fcebfdf6402a8876074a13c5708b [id=7a14b963398847d6b19313baae9772ea]) successfully deleted
I20250110 20:57:04.302999 5592 tablet_replica.cc:331] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:04.303707 5592 raft_consensus.cc:2238] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:04.304176 5592 raft_consensus.cc:2267] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.310406 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 0fd2cc66ab594f849f142d7847a1e0f5
I20250110 20:57:04.312234 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd2cc66ab594f849f142d7847a1e0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.312960 5592 ts_tablet_manager.cc:1905] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.318236 5594 ts_tablet_manager.cc:1428] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.112s user 0.005s sys 0.000s
I20250110 20:57:04.319146 5594 tablet_bootstrap.cc:492] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:04.326460 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd2cc66ab594f849f142d7847a1e0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:04.332008 5594 tablet_bootstrap.cc:654] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.334975 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 472174c6885b446a839721157914f047 (DEFAULT_TABLE table=default.Table_64d7746346a94643bdf6debc10a1ac6e [id=95036a214c1740d4b8391e06f7110688]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.336161 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472174c6885b446a839721157914f047. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.229301 5152 tablet_service.cc:1467] Processing CreateTablet for tablet e38b659f218d42298cb36752b5ca603d (DEFAULT_TABLE table=default.Table_13547221ce544ee7bf2f209caac6d5d8 [id=ba089f4f57284f80ac3621a88610fec1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:04.338045 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e38b659f218d42298cb36752b5ca603d consensus state has no leader
I20250110 20:57:04.338426 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e38b659f218d42298cb36752b5ca603d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.346848 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:04.345764 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_5745593bd7e54c27b24d7c06ebfc61b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:04.349025 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5745593bd7e54c27b24d7c06ebfc61b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.349709 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_24d07a4cb95a4deda26177c1bc951dc9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:04.350653 5155 tablet_service.cc:1467] Processing CreateTablet for tablet a7e7ffb063594128ba137f9aea86bf15 (DEFAULT_TABLE table=default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:04.351316 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24d07a4cb95a4deda26177c1bc951dc9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.352193 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 9155a626d26541ce9d0cf31508f19f15 (DEFAULT_TABLE table=default.Table_ec98243c2d784adfb3ab2a95bd2d89b8 [id=62486535b69d4a4798776d26dfdce501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.353639 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 5165408eb95e4849886252671dfb18d4 (DEFAULT_TABLE table=default.Table_1367656b1bef487abdee0e294f835b5f [id=ef270c5c9b0d465b8333565e19edd86b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.354615 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet 5165408eb95e4849886252671dfb18d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:04.358070 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 (DEFAULT_TABLE table=default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.349395 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fd2cc66ab594f849f142d7847a1e0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:04.360239 5593 tablet_replica.cc:331] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:04.360997 5593 raft_consensus.cc:2238] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:04.361742 5593 pending_rounds.cc:62] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:04.362015 5593 pending_rounds.cc:69] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878584613261312 op_type: NO_OP noop_request { }
I20250110 20:57:04.362469 5593 raft_consensus.cc:2883] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:04.362825 5593 raft_consensus.cc:2267] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.365686 5593 ts_tablet_manager.cc:1905] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.366055 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7e7ffb063594128ba137f9aea86bf15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.372263 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9155a626d26541ce9d0cf31508f19f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.371882 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_8df7bbf6722249b2be68514a607459b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:04.373482 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8df7bbf6722249b2be68514a607459b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.377688 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5165408eb95e4849886252671dfb18d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.385664 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ac7d62f978433dad004f2908c4bbc4. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:04.392616 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet e7bc3aae47f54c618798ff30c9cb75e8 because the tablet was not found. No further retry: Not found: Tablet not found: e7bc3aae47f54c618798ff30c9cb75e8
W20250110 20:57:04.393280 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 5165408eb95e4849886252671dfb18d4 because the tablet was not found. No further retry: Not found: Tablet not found: 5165408eb95e4849886252671dfb18d4
I20250110 20:57:04.398335 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 477b09061a4a4f278720168872a83258 (DEFAULT_TABLE table=default.Table_e570881031794b9ea4fd25d7b6fd4198 [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.399382 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 477b09061a4a4f278720168872a83258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.401052 4891 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:04.402135 5301 tablet_service.cc:1467] Processing CreateTablet for tablet e27e43461251413b97721323cd935374 (DEFAULT_TABLE table=default.Table_1d485a05ae354e4492c99bf9c67d6913 [id=729d5b2683d04c36ac582ea2965304de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.403098 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27e43461251413b97721323cd935374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.403831 4899 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_c97acba407484513a151d33ba86bd628" } new_table_name: "default.Table_682824c261c6472dbff5377598f241af" modify_external_catalogs: true
I20250110 20:57:04.405742 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 5793d909c6364047abc85ea5f993c521 (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.406164 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet b26f58b066f646e5b11ddf20cbbf9e1a from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:04.407269 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5793d909c6364047abc85ea5f993c521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.407122 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 5256d081843a492d8081c4319f3446d5 (DEFAULT_TABLE table=default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.408172 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5256d081843a492d8081c4319f3446d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.409251 4892 master_service.cc:946] ReplaceTablet: received request to replace tablet 29599cfce9ba4875b7aed08f5094ce21 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:04.410296 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 04fd9785888141abaab655f0ba74efb9 (DEFAULT_TABLE table=default.Table_18d27dbc0aca402787c14b0044972cb4 [id=b4a25185aa624d20a27bb31f0038436d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.411370 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fd9785888141abaab655f0ba74efb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.415294 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 477b09061a4a4f278720168872a83258 (DEFAULT_TABLE table=default.Table_e570881031794b9ea4fd25d7b6fd4198 [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.422078 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 477b09061a4a4f278720168872a83258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.443110 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_931826e8e40a4d04a1382a2816608f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:04.448666 5149 tablet_service.cc:1467] Processing CreateTablet for tablet e27e43461251413b97721323cd935374 (DEFAULT_TABLE table=default.Table_1d485a05ae354e4492c99bf9c67d6913 [id=729d5b2683d04c36ac582ea2965304de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.450179 5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27e43461251413b97721323cd935374. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:04.450610 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_931826e8e40a4d04a1382a2816608f99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.467518 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 5793d909c6364047abc85ea5f993c521 (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.469022 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5793d909c6364047abc85ea5f993c521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.481292 5147 tablet_service.cc:1467] Processing CreateTablet for tablet 04fd9785888141abaab655f0ba74efb9 (DEFAULT_TABLE table=default.Table_18d27dbc0aca402787c14b0044972cb4 [id=b4a25185aa624d20a27bb31f0038436d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.483304 5147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fd9785888141abaab655f0ba74efb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.493764 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_213a26b346b94108a30a80c719397f12" } modify_external_catalogs: true
I20250110 20:57:04.494380 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_213a26b346b94108a30a80c719397f12" } modify_external_catalogs: true
I20250110 20:57:04.494318 5146 tablet_service.cc:1467] Processing CreateTablet for tablet 5256d081843a492d8081c4319f3446d5 (DEFAULT_TABLE table=default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.495381 5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5256d081843a492d8081c4319f3446d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.497334 5606 raft_consensus.cc:491] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:04.497792 5606 raft_consensus.cc:513] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.500116 5606 leader_election.cc:290] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:04.501564 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637dda10276947d5842ed24bbf044d0d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:04.502189 5323 raft_consensus.cc:2463] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:04.503268 5111 leader_election.cc:304] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:04.509560 5606 raft_consensus.cc:2798] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:04.509889 5606 raft_consensus.cc:491] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:04.510185 5606 raft_consensus.cc:3054] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:04.526806 4894 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5745593bd7e54c27b24d7c06ebfc61b3" } modify_external_catalogs: true
I20250110 20:57:04.527333 4894 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5745593bd7e54c27b24d7c06ebfc61b3" } modify_external_catalogs: true
I20250110 20:57:04.542006 4891 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_635ff4f15ebf4d6580736411eaa6b821" } new_table_name: "default.Table_4a2a0aff7e744f57bc2c98b76bb0a199" modify_external_catalogs: true
W20250110 20:57:04.547060 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.252:37233, user_credentials={real_user=slave}} blocked reactor thread for 75960.9us
I20250110 20:57:04.524914 5606 raft_consensus.cc:513] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.558068 5600 tablet_bootstrap.cc:492] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.558221 5665 raft_consensus.cc:491] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:04.558440 5600 ts_tablet_manager.cc:1397] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.364s user 0.040s sys 0.031s
I20250110 20:57:04.558657 5665 raft_consensus.cc:513] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.560369 5600 raft_consensus.cc:357] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.561029 5600 raft_consensus.cc:383] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.561256 5600 raft_consensus.cc:738] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.561816 5600 consensus_queue.cc:260] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.564051 5665 leader_election.cc:290] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:04.565344 5606 leader_election.cc:290] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:04.577818 5612 raft_consensus.cc:491] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:04.578222 5612 raft_consensus.cc:513] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.579962 5612 leader_election.cc:290] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:04.585716 5600 ts_tablet_manager.cc:1428] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250110 20:57:04.590941 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:04.591553 5323 raft_consensus.cc:2463] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:04.593988 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.595902 5600 tablet_bootstrap.cc:492] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.619110 5600 tablet_bootstrap.cc:654] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.639245 5600 tablet_bootstrap.cc:492] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.639730 5600 ts_tablet_manager.cc:1397] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.000s
I20250110 20:57:04.641005 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_bf82fe64682446fa913015c01a1478fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:04.641290 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d6cd02ab71b14c2ab609b021be613df9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:04.642668 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf82fe64682446fa913015c01a1478fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.643759 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d29551acec544d57b14d18fecbf2722e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:04.645272 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d29551acec544d57b14d18fecbf2722e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:04.647665 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6cd02ab71b14c2ab609b021be613df9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:04.654451 5600 raft_consensus.cc:357] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.656982 5600 raft_consensus.cc:383] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.657473 5600 raft_consensus.cc:738] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.658090 5600 consensus_queue.cc:260] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.658396 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637dda10276947d5842ed24bbf044d0d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:04.658989 5323 raft_consensus.cc:3054] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:04.660907 5600 ts_tablet_manager.cc:1428] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20250110 20:57:04.661684 5600 tablet_bootstrap.cc:492] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.670244 5323 raft_consensus.cc:2463] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:04.676249 5600 tablet_bootstrap.cc:654] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.680653 5597 tablet_bootstrap.cc:654] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:04.683853 4899 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b26f58b066f646e5b11ddf20cbbf9e1a consensus state has no leader
I20250110 20:57:04.689440 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet b26f58b066f646e5b11ddf20cbbf9e1a
I20250110 20:57:04.691179 4896 catalog_manager.cc:6596] ReplaceTablet: tablet b26f58b066f646e5b11ddf20cbbf9e1a deleted and replaced by tablet 78538b2e7bcc45c7afaa145bad6e5813
I20250110 20:57:04.581843 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:04.695724 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 78538b2e7bcc45c7afaa145bad6e5813) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:04.698729 4899 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:04.704506 5600 tablet_bootstrap.cc:492] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.707516 5600 ts_tablet_manager.cc:1397] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.004s
I20250110 20:57:04.709766 4899 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
W20250110 20:57:04.712863 4887 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e38b659f218d42298cb36752b5ca603d (table default.Table_635ff4f15ebf4d6580736411eaa6b821 [id=ba089f4f57284f80ac3621a88610fec1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e38b659f218d42298cb36752b5ca603d consensus state has no leader
I20250110 20:57:04.709822 5600 raft_consensus.cc:357] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.715875 5600 raft_consensus.cc:383] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.716141 5600 raft_consensus.cc:738] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.716235 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 78538b2e7bcc45c7afaa145bad6e5813) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:04.717773 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:04.716760 5600 consensus_queue.cc:260] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:04.720566 5255 leader_election.cc:343] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.724164 5600 ts_tablet_manager.cc:1428] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20250110 20:57:04.728719 5438 tablet_service.cc:1467] Processing CreateTablet for tablet 477b09061a4a4f278720168872a83258 (DEFAULT_TABLE table=default.Table_e570881031794b9ea4fd25d7b6fd4198 [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.730479 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 477b09061a4a4f278720168872a83258. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.735616 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:04.739338 5600 tablet_bootstrap.cc:492] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.738899 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637dda10276947d5842ed24bbf044d0d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:04.737507 5175 raft_consensus.cc:2463] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:04.736800 5111 leader_election.cc:304] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:04.745234 5600 tablet_bootstrap.cc:654] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:04.745142 5110 leader_election.cc:343] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.741333 5437 tablet_service.cc:1467] Processing CreateTablet for tablet e27e43461251413b97721323cd935374 (DEFAULT_TABLE table=default.Table_1d485a05ae354e4492c99bf9c67d6913 [id=729d5b2683d04c36ac582ea2965304de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.747998 5257 leader_election.cc:304] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:04.748703 5612 raft_consensus.cc:2798] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:04.749079 5612 raft_consensus.cc:491] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:04.749233 5606 raft_consensus.cc:2798] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:04.749389 5612 raft_consensus.cc:3054] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:04.749567 5606 raft_consensus.cc:491] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:04.749869 5606 raft_consensus.cc:3054] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:04.753793 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 62136.6us
I20250110 20:57:04.754519 5111 leader_election.cc:304] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:04.749295 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637dda10276947d5842ed24bbf044d0d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:04.756996 5612 raft_consensus.cc:513] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.757714 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e27e43461251413b97721323cd935374. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.759814 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
W20250110 20:57:04.761878 5255 leader_election.cc:343] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:04.755627 5110 leader_election.cc:343] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:04.766686 5110 leader_election.cc:343] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.767503 5612 leader_election.cc:290] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:04.767791 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:04.757091 5606 raft_consensus.cc:513] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.769958 5606 leader_election.cc:290] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:04.769492 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 04fd9785888141abaab655f0ba74efb9 (DEFAULT_TABLE table=default.Table_18d27dbc0aca402787c14b0044972cb4 [id=b4a25185aa624d20a27bb31f0038436d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.770670 5175 raft_consensus.cc:2388] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 46005a5920cc4a3697a5c61b1c271384 in current term 1: Already voted for candidate ac9003be04f340f3886b5625cc928eb0 in this term.
I20250110 20:57:04.771409 5434 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 78538b2e7bcc45c7afaa145bad6e5813) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:04.773149 5257 leader_election.cc:304] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:04.773924 5612 raft_consensus.cc:2743] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 20:57:04.774801 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:04.775523 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:04.774042 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 5793d909c6364047abc85ea5f993c521 (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.776196 5323 raft_consensus.cc:2388] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ac9003be04f340f3886b5625cc928eb0 in current term 1: Already voted for candidate 46005a5920cc4a3697a5c61b1c271384 in this term.
I20250110 20:57:04.776924 5435 tablet_service.cc:1467] Processing CreateTablet for tablet 5256d081843a492d8081c4319f3446d5 (DEFAULT_TABLE table=default.Table_f7a60bd067b342fb8e7d1cd14f1a37ce [id=d89d02a8ce0e4bd0bf2561469fd627ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:04.778110 5600 tablet_bootstrap.cc:492] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.778468 5435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5256d081843a492d8081c4319f3446d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.778486 5600 ts_tablet_manager.cc:1397] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.005s
I20250110 20:57:04.781514 5606 raft_consensus.cc:2798] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:04.781606 5600 raft_consensus.cc:357] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.782312 5600 raft_consensus.cc:383] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.782219 5606 raft_consensus.cc:695] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:04.782568 5600 raft_consensus.cc:738] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.783218 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04fd9785888141abaab655f0ba74efb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.783149 5606 consensus_queue.cc:237] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.785703 5600 consensus_queue.cc:260] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.789949 5600 ts_tablet_manager.cc:1428] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250110 20:57:04.795161 5600 tablet_bootstrap.cc:492] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.803725 5600 tablet_bootstrap.cc:654] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.804633 5593 ts_tablet_manager.cc:1918] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:04.805084 5593 log.cc:1198] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/0fd2cc66ab594f849f142d7847a1e0f5
W20250110 20:57:04.805326 5110 leader_election.cc:343] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.806188 5110 leader_election.cc:304] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:04.806984 5665 raft_consensus.cc:2743] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 20:57:04.815361 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5793d909c6364047abc85ea5f993c521. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:04.816814 5600 tablet_bootstrap.cc:492] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.818358 5600 ts_tablet_manager.cc:1397] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.002s
I20250110 20:57:04.821019 5600 raft_consensus.cc:357] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.821765 5600 raft_consensus.cc:383] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.822041 5600 raft_consensus.cc:738] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.822626 5600 consensus_queue.cc:260] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.827000 5600 ts_tablet_manager.cc:1428] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250110 20:57:04.829028 5600 tablet_bootstrap.cc:492] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.830996 5597 tablet_bootstrap.cc:492] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:04.831367 5597 ts_tablet_manager.cc:1397] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.616s user 0.008s sys 0.004s
I20250110 20:57:04.835606 5600 tablet_bootstrap.cc:654] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.826576 5673 raft_consensus.cc:491] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:04.841650 5673 raft_consensus.cc:513] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.844288 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "332dfe3a0e45468695aeb7d7eb7c30ca" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
W20250110 20:57:04.845575 5398 leader_election.cc:343] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:04.846761 5593 ts_tablet_manager.cc:1939] T 0fd2cc66ab594f849f142d7847a1e0f5 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:04.855937 5600 tablet_bootstrap.cc:492] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.856381 5600 ts_tablet_manager.cc:1397] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.017s
I20250110 20:57:04.857391 5673 leader_election.cc:290] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:04.858853 5594 tablet_bootstrap.cc:492] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:04.859225 5594 ts_tablet_manager.cc:1397] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.540s user 0.005s sys 0.005s
I20250110 20:57:04.858958 5600 raft_consensus.cc:357] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.859671 5600 raft_consensus.cc:383] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.859922 5600 raft_consensus.cc:738] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.860486 5600 consensus_queue.cc:260] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.862324 5594 raft_consensus.cc:357] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.862732 5600 ts_tablet_manager.cc:1428] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20250110 20:57:04.862920 5594 raft_consensus.cc:383] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.863592 5600 tablet_bootstrap.cc:492] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.867244 5592 ts_tablet_manager.cc:1918] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:04.867600 5592 log.cc:1198] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/80a22306f7c7487d996969e349f09766
I20250110 20:57:04.868227 5592 ts_tablet_manager.cc:1939] T 80a22306f7c7487d996969e349f09766 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
W20250110 20:57:04.873001 5399 leader_election.cc:343] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: 332dfe3a0e45468695aeb7d7eb7c30ca
I20250110 20:57:04.873234 5600 tablet_bootstrap.cc:654] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.873641 5399 leader_election.cc:304] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:04.874701 5673 raft_consensus.cc:2743] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:04.874743 4887 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 -> Peer 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233): Couldn't send request to peer 14d1e2d5c72f498f915c367e8750361f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250110 20:57:04.881145 5612 raft_consensus.cc:491] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:04.881565 5612 raft_consensus.cc:513] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.863143 5594 raft_consensus.cc:738] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.883304 5612 leader_election.cc:290] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:04.885634 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b5876f954e469f86498a529e5c4a68" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:04.883211 5594 consensus_queue.cc:260] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.885118 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b5876f954e469f86498a529e5c4a68" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:04.886682 5255 leader_election.cc:343] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.892225 5594 ts_tablet_manager.cc:1428] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.033s user 0.003s sys 0.002s
I20250110 20:57:04.893430 5594 tablet_bootstrap.cc:492] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:04.896215 5600 tablet_bootstrap.cc:492] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.896625 5600 ts_tablet_manager.cc:1397] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.002s
W20250110 20:57:04.898262 5257 leader_election.cc:343] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:04.898623 5257 leader_election.cc:304] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:04.898855 4896 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e570881031794b9ea4fd25d7b6fd4198" } new_table_name: "default.Table_44e92821147643aa9a6b725eadd88b39" modify_external_catalogs: true
I20250110 20:57:04.899001 5600 raft_consensus.cc:357] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.899523 5612 raft_consensus.cc:2743] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:04.899749 5600 raft_consensus.cc:383] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.900017 5600 raft_consensus.cc:738] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.900548 5600 consensus_queue.cc:260] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.902319 5600 ts_tablet_manager.cc:1428] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20250110 20:57:04.903381 5600 tablet_bootstrap.cc:492] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.908368 5600 tablet_bootstrap.cc:654] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:04.909255 4891 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e38b659f218d42298cb36752b5ca603d consensus state has no leader
I20250110 20:57:04.912307 4894 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet d8d95608744746608a6392861453668d
I20250110 20:57:04.927182 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]). Expected version 3 got 0
I20250110 20:57:04.935526 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 0fd2cc66ab594f849f142d7847a1e0f5 (table default.Table_14adb01640aa40d699cfb94980d7aea3 [id=1c2a6b50e68e4363b5ebf736451866a6]) successfully deleted
W20250110 20:57:04.936527 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet b26f58b066f646e5b11ddf20cbbf9e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b26f58b066f646e5b11ddf20cbbf9e1a already in progress: creating tablet
I20250110 20:57:04.936803 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]). Expected version 3 got 0
I20250110 20:57:04.938030 5597 raft_consensus.cc:357] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.938793 5597 raft_consensus.cc:383] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.939095 5597 raft_consensus.cc:738] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.939814 5597 consensus_queue.cc:260] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.947247 5600 tablet_bootstrap.cc:492] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:04.947631 5600 ts_tablet_manager.cc:1397] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.044s user 0.010s sys 0.011s
I20250110 20:57:04.948369 5612 raft_consensus.cc:491] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:04.948824 5612 raft_consensus.cc:513] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:04.951095 5612 leader_election.cc:290] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:04.950956 5600 raft_consensus.cc:357] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.951671 5600 raft_consensus.cc:383] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:04.951942 5600 raft_consensus.cc:738] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:04.952495 5600 consensus_queue.cc:260] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:04.952929 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 80a22306f7c7487d996969e349f09766 (table default.Table_e8fd956943c647e1890ffe3f3d3597f7 [id=3fb5402eb3a944d989a916f70a71c2d3]) successfully deleted
W20250110 20:57:04.953529 5257 leader_election.cc:343] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: e7bc3aae47f54c618798ff30c9cb75e8
I20250110 20:57:04.955323 5600 ts_tablet_manager.cc:1428] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20250110 20:57:04.956632 5600 tablet_bootstrap.cc:492] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:04.961445 5595 ts_tablet_manager.cc:1918] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:04.961611 5600 tablet_bootstrap.cc:654] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.962011 5595 log.cc:1198] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/332dfe3a0e45468695aeb7d7eb7c30ca
I20250110 20:57:04.962487 5592 tablet_replica.cc:331] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:04.963156 5592 raft_consensus.cc:2238] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:04.963586 5592 raft_consensus.cc:2267] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:04.954835 5594 tablet_bootstrap.cc:654] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:04.965548 4894 catalog_manager.cc:5526] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:04.967181 4894 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]). Expected version 3 got 0
I20250110 20:57:04.977871 5592 ts_tablet_manager.cc:1905] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:04.983117 5594 tablet_bootstrap.cc:492] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:04.983487 5594 ts_tablet_manager.cc:1397] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.000s
I20250110 20:57:04.993788 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 22b5876f954e469f86498a529e5c4a68
W20250110 20:57:04.995219 4896 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 477b09061a4a4f278720168872a83258 consensus state has no leader
I20250110 20:57:04.995988 4892 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 29599cfce9ba4875b7aed08f5094ce21
I20250110 20:57:04.997243 4892 catalog_manager.cc:6596] ReplaceTablet: tablet 29599cfce9ba4875b7aed08f5094ce21 deleted and replaced by tablet d9776df98cb54be38a521852634490e7
I20250110 20:57:05.006590 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5876f954e469f86498a529e5c4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.008844 5595 ts_tablet_manager.cc:1939] T 332dfe3a0e45468695aeb7d7eb7c30ca P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.011297 5595 tablet_replica.cc:331] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:05.011583 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 332dfe3a0e45468695aeb7d7eb7c30ca (table default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]) successfully deleted
I20250110 20:57:05.012046 5595 raft_consensus.cc:2238] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.012466 5595 raft_consensus.cc:2267] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.013146 4892 master_service.cc:946] ReplaceTablet: received request to replace tablet 477b09061a4a4f278720168872a83258 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:05.013989 5435 tablet_service.cc:1514] Processing DeleteTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 with delete_type TABLET_DATA_DELETED (replaced by tablet d9776df98cb54be38a521852634490e7) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.014168 5597 ts_tablet_manager.cc:1428] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.183s user 0.005s sys 0.002s
I20250110 20:57:05.014919 5597 tablet_bootstrap.cc:492] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:05.016175 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 with delete_type TABLET_DATA_DELETED (replaced by tablet d9776df98cb54be38a521852634490e7) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.016963 5437 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5876f954e469f86498a529e5c4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.018620 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7bc3aae47f54c618798ff30c9cb75e8" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:05.022092 5255 leader_election.cc:343] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.022477 5255 leader_election.cc:304] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:05.023454 5612 raft_consensus.cc:2743] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:05.024993 5594 raft_consensus.cc:357] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.039459 5600 tablet_bootstrap.cc:492] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:05.039882 5600 ts_tablet_manager.cc:1397] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.083s user 0.030s sys 0.037s
I20250110 20:57:05.042819 5594 raft_consensus.cc:383] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.043843 5594 raft_consensus.cc:738] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.044536 5595 ts_tablet_manager.cc:1905] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.050628 5600 raft_consensus.cc:357] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.051291 5600 raft_consensus.cc:383] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.051541 5600 raft_consensus.cc:738] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.052268 5600 consensus_queue.cc:260] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.061736 5295 tablet_service.cc:1467] Processing CreateTablet for tablet ae5956b1aa17449497f9695a111ebb66 (DEFAULT_TABLE table=default.Table_24d07a4cb95a4deda26177c1bc951dc9 [id=6876fd7b8dd8452b94c04eac6be82cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.062544 5600 ts_tablet_manager.cc:1428] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250110 20:57:05.064419 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5956b1aa17449497f9695a111ebb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.065339 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 3a520230d9fd4a74914e46f784dcf747 (DEFAULT_TABLE table=default.Table_8df7bbf6722249b2be68514a607459b9 [id=64cfe285ff2041778f15cf7e6020e9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.067853 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.074275 5600 tablet_bootstrap.cc:492] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:05.078781 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78538b2e7bcc45c7afaa145bad6e5813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.079190 5600 tablet_bootstrap.cc:654] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.084264 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a520230d9fd4a74914e46f784dcf747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.087641 4892 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 477b09061a4a4f278720168872a83258
I20250110 20:57:05.093389 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet 477b09061a4a4f278720168872a83258 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.093698 4892 catalog_manager.cc:6596] ReplaceTablet: tablet 477b09061a4a4f278720168872a83258 deleted and replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:05.097805 5595 ts_tablet_manager.cc:1918] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.068312 5435 tablet_service.cc:1467] Processing CreateTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.099109 5595 log.cc:1198] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/5165408eb95e4849886252671dfb18d4
I20250110 20:57:05.099411 5435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78538b2e7bcc45c7afaa145bad6e5813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.102970 5595 ts_tablet_manager.cc:1939] T 5165408eb95e4849886252671dfb18d4 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.089871 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 477b09061a4a4f278720168872a83258 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.087549 5437 tablet_service.cc:1467] Processing CreateTablet for tablet ae5956b1aa17449497f9695a111ebb66 (DEFAULT_TABLE table=default.Table_24d07a4cb95a4deda26177c1bc951dc9 [id=6876fd7b8dd8452b94c04eac6be82cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.106237 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5956b1aa17449497f9695a111ebb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.107810 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 5165408eb95e4849886252671dfb18d4 (table default.Table_1367656b1bef487abdee0e294f835b5f [id=ef270c5c9b0d465b8333565e19edd86b]) successfully deleted
I20250110 20:57:05.109623 5595 tablet_replica.cc:331] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:05.110587 5147 tablet_service.cc:1467] Processing CreateTablet for tablet 3a520230d9fd4a74914e46f784dcf747 (DEFAULT_TABLE table=default.Table_8df7bbf6722249b2be68514a607459b9 [id=64cfe285ff2041778f15cf7e6020e9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.111877 5595 raft_consensus.cc:2238] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.112113 5147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a520230d9fd4a74914e46f784dcf747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.112835 5595 raft_consensus.cc:2267] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.113523 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_8e63a6ded00f470d94804f6f15343b29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:05.113641 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ba448520fdb24cd1be60910f20647848"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:05.115074 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e63a6ded00f470d94804f6f15343b29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:05.115219 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba448520fdb24cd1be60910f20647848 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.116420 5595 ts_tablet_manager.cc:1905] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.116503 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d2f92229bca74d26baf2508b363c90ad" } modify_external_catalogs: true
I20250110 20:57:05.117030 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d2f92229bca74d26baf2508b363c90ad" } modify_external_catalogs: true
I20250110 20:57:05.126325 5148 tablet_service.cc:1467] Processing CreateTablet for tablet ae5956b1aa17449497f9695a111ebb66 (DEFAULT_TABLE table=default.Table_24d07a4cb95a4deda26177c1bc951dc9 [id=6876fd7b8dd8452b94c04eac6be82cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.127571 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5956b1aa17449497f9695a111ebb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.135186 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 961fef96e9124aafb1c51e287f603b64 (DEFAULT_TABLE table=default.Table_d6cd02ab71b14c2ab609b021be613df9 [id=9c593f8a26874512a72f0bfc20975c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.136274 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961fef96e9124aafb1c51e287f603b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.137665 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5876f954e469f86498a529e5c4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.138926 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 with delete_type TABLET_DATA_DELETED (replaced by tablet d9776df98cb54be38a521852634490e7) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.139160 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_6d29a7293ace46b186d68a71d0a5a688"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:05.139726 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 477b09061a4a4f278720168872a83258 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:05.140730 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d29a7293ace46b186d68a71d0a5a688 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.140959 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.142444 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78538b2e7bcc45c7afaa145bad6e5813. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.144311 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_26dcb5fd161748d3897adbcc67f697e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:05.144788 5298 tablet_service.cc:1467] Processing CreateTablet for tablet 961fef96e9124aafb1c51e287f603b64 (DEFAULT_TABLE table=default.Table_d6cd02ab71b14c2ab609b021be613df9 [id=9c593f8a26874512a72f0bfc20975c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.145807 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961fef96e9124aafb1c51e287f603b64. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:05.145871 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26dcb5fd161748d3897adbcc67f697e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.146131 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 961fef96e9124aafb1c51e287f603b64 (DEFAULT_TABLE table=default.Table_d6cd02ab71b14c2ab609b021be613df9 [id=9c593f8a26874512a72f0bfc20975c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.147148 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 961fef96e9124aafb1c51e287f603b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.155305 5299 tablet_service.cc:1514] Processing DeleteTablet for tablet 5165408eb95e4849886252671dfb18d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.156635 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_cd3dd0abc91e4209b6f8586377dafc01" } new_table_name: "default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f" modify_external_catalogs: true
I20250110 20:57:05.159008 5595 ts_tablet_manager.cc:1918] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.164014 5150 tablet_service.cc:1467] Processing CreateTablet for tablet f1fafed70adb496d97acfd5fea356292 (DEFAULT_TABLE table=default.Table_bf82fe64682446fa913015c01a1478fe [id=e2dbf27ca6424662a7a14c9a9ae5587d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.165089 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1fafed70adb496d97acfd5fea356292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.164471 5299 tablet_service.cc:1467] Processing CreateTablet for tablet f1fafed70adb496d97acfd5fea356292 (DEFAULT_TABLE table=default.Table_bf82fe64682446fa913015c01a1478fe [id=e2dbf27ca6424662a7a14c9a9ae5587d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.166970 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1fafed70adb496d97acfd5fea356292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.168869 5149 tablet_service.cc:1467] Processing CreateTablet for tablet 19d6eca349514b9fb0fff7cd6230027c (DEFAULT_TABLE table=default.Table_931826e8e40a4d04a1382a2816608f99 [id=bb54800eb7624f4482ddee01f59494ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.169235 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 19d6eca349514b9fb0fff7cd6230027c (DEFAULT_TABLE table=default.Table_931826e8e40a4d04a1382a2816608f99 [id=bb54800eb7624f4482ddee01f59494ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.169860 5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d6eca349514b9fb0fff7cd6230027c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.170248 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d6eca349514b9fb0fff7cd6230027c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.182442 5151 tablet_service.cc:1467] Processing CreateTablet for tablet d9776df98cb54be38a521852634490e7 (DEFAULT_TABLE table=default.Table_cd3dd0abc91e4209b6f8586377dafc01 [id=3bbd1acd709b44899907148df944b20d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:05.179344 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 22b5876f954e469f86498a529e5c4a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22b5876f954e469f86498a529e5c4a68 already in progress: creating tablet
I20250110 20:57:05.183449 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9776df98cb54be38a521852634490e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:05.183454 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 29599cfce9ba4875b7aed08f5094ce21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29599cfce9ba4875b7aed08f5094ce21 already in progress: creating tablet
W20250110 20:57:05.184027 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 477b09061a4a4f278720168872a83258 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 477b09061a4a4f278720168872a83258 already in progress: creating tablet
I20250110 20:57:05.184271 5297 tablet_service.cc:1467] Processing CreateTablet for tablet d9776df98cb54be38a521852634490e7 (DEFAULT_TABLE table=default.Table_cd3dd0abc91e4209b6f8586377dafc01 [id=3bbd1acd709b44899907148df944b20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.185379 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9776df98cb54be38a521852634490e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.187213 5438 tablet_service.cc:1467] Processing CreateTablet for tablet f1fafed70adb496d97acfd5fea356292 (DEFAULT_TABLE table=default.Table_bf82fe64682446fa913015c01a1478fe [id=e2dbf27ca6424662a7a14c9a9ae5587d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.188266 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1fafed70adb496d97acfd5fea356292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.190068 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 19d6eca349514b9fb0fff7cd6230027c (DEFAULT_TABLE table=default.Table_931826e8e40a4d04a1382a2816608f99 [id=bb54800eb7624f4482ddee01f59494ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.191032 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d6eca349514b9fb0fff7cd6230027c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.193538 5606 raft_consensus.cc:491] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.194067 5441 tablet_service.cc:1467] Processing CreateTablet for tablet d9776df98cb54be38a521852634490e7 (DEFAULT_TABLE table=default.Table_cd3dd0abc91e4209b6f8586377dafc01 [id=3bbd1acd709b44899907148df944b20d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.194006 5606 raft_consensus.cc:513] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.194999 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9776df98cb54be38a521852634490e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.196164 5606 leader_election.cc:290] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:05.198812 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe93f1a08f3432dba9c6ce76b4c99b0" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:05.199393 5323 raft_consensus.cc:2463] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:05.200886 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe93f1a08f3432dba9c6ce76b4c99b0" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:05.201979 5612 raft_consensus.cc:491] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.202378 5612 raft_consensus.cc:513] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.204111 5612 leader_election.cc:290] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:05.211308 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 31b65488c884452d876ced1c3d1efb9a (DEFAULT_TABLE table=default.Table_d29551acec544d57b14d18fecbf2722e [id=16478fe1cba54c88ba8245c365d5a992]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.217783 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31b65488c884452d876ced1c3d1efb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.218124 5597 tablet_bootstrap.cc:654] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.219651 5111 leader_election.cc:304] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:05.220453 5606 raft_consensus.cc:2798] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:05.220338 4891 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_18d27dbc0aca402787c14b0044972cb4" } new_table_name: "default.Table_b5ff248226c7469494af3086db1957fe" modify_external_catalogs: true
I20250110 20:57:05.220767 5606 raft_consensus.cc:491] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:05.221037 5606 raft_consensus.cc:3054] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:05.226147 5110 leader_election.cc:343] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.227157 5595 log.cc:1198] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/0fd2cc66ab594f849f142d7847a1e0f5
I20250110 20:57:05.227769 5595 ts_tablet_manager.cc:1939] T 0fd2cc66ab594f849f142d7847a1e0f5 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.227861 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26f58b066f646e5b11ddf20cbbf9e1a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:05.237988 5600 tablet_bootstrap.cc:492] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:05.238360 5600 ts_tablet_manager.cc:1397] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.168s user 0.018s sys 0.012s
I20250110 20:57:05.240469 5600 raft_consensus.cc:357] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:05.241096 5600 raft_consensus.cc:383] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.241330 5600 raft_consensus.cc:738] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.241299 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d29551acec544d57b14d18fecbf2722e" } modify_external_catalogs: true
I20250110 20:57:05.241923 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d29551acec544d57b14d18fecbf2722e" } modify_external_catalogs: true
I20250110 20:57:05.241926 5600 consensus_queue.cc:260] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:05.246214 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 0fd2cc66ab594f849f142d7847a1e0f5 (table default.Table_14adb01640aa40d699cfb94980d7aea3 [id=1c2a6b50e68e4363b5ebf736451866a6]) successfully deleted
I20250110 20:57:05.246412 5600 ts_tablet_manager.cc:1428] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250110 20:57:05.247802 5595 tablet_replica.cc:331] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:05.248461 5595 raft_consensus.cc:2238] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.248838 5595 raft_consensus.cc:2267] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.249626 5600 tablet_bootstrap.cc:492] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:05.253762 5257 leader_election.cc:343] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.256660 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e7ffb063594128ba137f9aea86bf15 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.258344 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 31b65488c884452d876ced1c3d1efb9a (DEFAULT_TABLE table=default.Table_d29551acec544d57b14d18fecbf2722e [id=16478fe1cba54c88ba8245c365d5a992]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.259903 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31b65488c884452d876ced1c3d1efb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.263522 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26f58b066f646e5b11ddf20cbbf9e1a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:05.264603 5255 leader_election.cc:343] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.265009 5255 leader_election.cc:304] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:05.265956 5612 raft_consensus.cc:2743] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:05.268373 5595 ts_tablet_manager.cc:1905] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.274472 5606 raft_consensus.cc:513] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.275735 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 2dc0d3abef5741339fdbb9abeed20b15
I20250110 20:57:05.289120 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc0d3abef5741339fdbb9abeed20b15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.290138 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe93f1a08f3432dba9c6ce76b4c99b0" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:05.290704 5323 raft_consensus.cc:3054] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:05.293978 5606 leader_election.cc:290] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:05.295105 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fe93f1a08f3432dba9c6ce76b4c99b0" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
W20250110 20:57:05.296260 5110 leader_election.cc:343] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.296949 5612 raft_consensus.cc:491] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.297430 5612 raft_consensus.cc:513] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.299229 5612 leader_election.cc:290] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:05.299614 5612 raft_consensus.cc:491] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.299983 5612 raft_consensus.cc:513] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:05.310577 5597 tablet_bootstrap.cc:492] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:05.310593 5323 raft_consensus.cc:2463] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:05.311108 5597 ts_tablet_manager.cc:1397] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.296s user 0.006s sys 0.010s
I20250110 20:57:05.312739 5612 leader_election.cc:290] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:05.313285 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc0d3abef5741339fdbb9abeed20b15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.313716 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 31b65488c884452d876ced1c3d1efb9a (DEFAULT_TABLE table=default.Table_d29551acec544d57b14d18fecbf2722e [id=16478fe1cba54c88ba8245c365d5a992]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.313474 5597 raft_consensus.cc:357] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.315021 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31b65488c884452d876ced1c3d1efb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.315557 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 2dc0d3abef5741339fdbb9abeed20b15 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.316854 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29599cfce9ba4875b7aed08f5094ce21" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:05.318250 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e7ffb063594128ba137f9aea86bf15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:05.319218 5255 leader_election.cc:343] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:05.320026 5255 leader_election.cc:343] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.098053 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 3a520230d9fd4a74914e46f784dcf747 (DEFAULT_TABLE table=default.Table_8df7bbf6722249b2be68514a607459b9 [id=64cfe285ff2041778f15cf7e6020e9cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.321738 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a520230d9fd4a74914e46f784dcf747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.328022 5600 tablet_bootstrap.cc:654] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.048170 5594 consensus_queue.cc:260] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.331358 5594 ts_tablet_manager.cc:1428] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.348s user 0.007s sys 0.000s
I20250110 20:57:05.332158 5594 tablet_bootstrap.cc:492] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:05.338641 5595 ts_tablet_manager.cc:1918] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.338738 4899 master_service.cc:946] ReplaceTablet: received request to replace tablet 637dda10276947d5842ed24bbf044d0d from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:05.338994 5595 log.cc:1198] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/e7bc3aae47f54c618798ff30c9cb75e8
I20250110 20:57:05.339561 5595 ts_tablet_manager.cc:1939] T e7bc3aae47f54c618798ff30c9cb75e8 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.340029 5593 tablet_replica.cc:331] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:05.346021 5593 raft_consensus.cc:2238] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.346429 5593 raft_consensus.cc:2267] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.348006 5593 ts_tablet_manager.cc:1905] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:05.349849 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9776df98cb54be38a521852634490e7 consensus state has no leader
I20250110 20:57:05.354869 5597 raft_consensus.cc:383] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:05.358036 4891 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04fd9785888141abaab655f0ba74efb9 consensus state has no leader
I20250110 20:57:05.358716 5597 raft_consensus.cc:738] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.359185 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:05.359254 5597 consensus_queue.cc:260] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.372860 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b65488c884452d876ced1c3d1efb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.378830 5597 ts_tablet_manager.cc:1428] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.067s user 0.004s sys 0.001s
I20250110 20:57:05.379643 5597 tablet_bootstrap.cc:492] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:05.383249 5461 raft_consensus.cc:3054] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:05.394773 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b65488c884452d876ced1c3d1efb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.394958 4891 master_service.cc:946] ReplaceTablet: received request to replace tablet 961fef96e9124aafb1c51e287f603b64 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:05.396643 5593 ts_tablet_manager.cc:1918] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.397007 5593 log.cc:1198] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/2dc0d3abef5741339fdbb9abeed20b15
I20250110 20:57:05.397482 5461 raft_consensus.cc:1270] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Refusing update from remote peer ac9003be04f340f3886b5625cc928eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:05.397686 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 with delete_type TABLET_DATA_DELETED (replaced by tablet d9776df98cb54be38a521852634490e7) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.400434 5606 consensus_queue.cc:1035] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:57:05.415865 5597 tablet_bootstrap.cc:654] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.417091 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_56fa755492734855973131b69735b169"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:05.418540 5593 ts_tablet_manager.cc:1939] T 2dc0d3abef5741339fdbb9abeed20b15 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
W20250110 20:57:05.420116 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56fa755492734855973131b69735b169 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.436108 5593 tablet_replica.cc:331] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:05.436426 5595 tablet_replica.cc:331] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:05.436746 5593 raft_consensus.cc:2238] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.437158 5593 raft_consensus.cc:2267] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.447693 5593 ts_tablet_manager.cc:1905] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.420531 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 2dc0d3abef5741339fdbb9abeed20b15 (table default.Table_d2f92229bca74d26baf2508b363c90ad [id=9daaee4eb9574713b7914a551680c52b]) successfully deleted
I20250110 20:57:05.444312 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet e7bc3aae47f54c618798ff30c9cb75e8 (table default.Table_ccaf097e265d4ed6a3ed2e9c0984229a [id=af9522c35cd74c119bdc1af28f7bab6c]) successfully deleted
I20250110 20:57:05.449396 5595 raft_consensus.cc:2238] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.449843 5595 raft_consensus.cc:2267] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.449889 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b65488c884452d876ced1c3d1efb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.450558 5597 tablet_bootstrap.cc:492] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:05.450939 5597 ts_tablet_manager.cc:1397] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.001s
I20250110 20:57:05.451370 5595 ts_tablet_manager.cc:1905] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.457185 5689 raft_consensus.cc:491] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.457316 5111 leader_election.cc:304] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:05.457590 5689 raft_consensus.cc:513] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.458523 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29599cfce9ba4875b7aed08f5094ce21" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:05.459421 5689 leader_election.cc:290] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:05.460146 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e7ffb063594128ba137f9aea86bf15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:05.460994 5606 raft_consensus.cc:2798] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 20:57:05.461449 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 31b65488c884452d876ced1c3d1efb9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31b65488c884452d876ced1c3d1efb9a already in progress: creating tablet
I20250110 20:57:05.462160 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574171c94ca6450daf9fb608dbe9b002" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:05.463934 5323 raft_consensus.cc:1270] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Refusing update from remote peer ac9003be04f340f3886b5625cc928eb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250110 20:57:05.464656 5255 leader_election.cc:343] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:05.463747 5257 leader_election.cc:343] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:05.465514 5257 leader_election.cc:304] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:05.466923 5689 raft_consensus.cc:2743] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:05.468392 5257 leader_election.cc:343] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.468737 5257 leader_election.cc:304] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:05.477548 5689 raft_consensus.cc:2743] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:05.481505 5600 tablet_bootstrap.cc:492] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:05.481865 5600 ts_tablet_manager.cc:1397] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.232s user 0.009s sys 0.002s
I20250110 20:57:05.484120 5600 raft_consensus.cc:357] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.484265 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_bb34f3dffd1d417690819599206a7a92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:05.484845 5600 raft_consensus.cc:383] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.485101 5600 raft_consensus.cc:738] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.485904 5600 consensus_queue.cc:260] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.488119 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_bb5cbd6c878c4302aa43b01c0f1c3e96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:05.489017 5600 ts_tablet_manager.cc:1428] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
W20250110 20:57:05.489692 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb5cbd6c878c4302aa43b01c0f1c3e96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.490888 5600 tablet_bootstrap.cc:492] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:05.492632 5595 ts_tablet_manager.cc:1918] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.492969 5595 log.cc:1198] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/b26f58b066f646e5b11ddf20cbbf9e1a
I20250110 20:57:05.493566 5595 ts_tablet_manager.cc:1939] T b26f58b066f646e5b11ddf20cbbf9e1a P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
W20250110 20:57:05.500392 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb34f3dffd1d417690819599206a7a92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.503718 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_4a2a0aff7e744f57bc2c98b76bb0a199" } modify_external_catalogs: true
I20250110 20:57:05.504269 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_4a2a0aff7e744f57bc2c98b76bb0a199" } modify_external_catalogs: true
I20250110 20:57:05.505313 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet b26f58b066f646e5b11ddf20cbbf9e1a (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]) successfully deleted
I20250110 20:57:05.506232 5595 tablet_replica.cc:331] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:05.506871 5595 raft_consensus.cc:2238] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.507258 5595 raft_consensus.cc:2267] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.507745 5600 tablet_bootstrap.cc:654] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.508656 5595 ts_tablet_manager.cc:1905] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.513989 5697 consensus_queue.cc:1035] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250110 20:57:05.517493 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 54703us
I20250110 20:57:05.518615 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574171c94ca6450daf9fb608dbe9b002" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:05.519140 5176 raft_consensus.cc:2463] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:05.520188 5257 leader_election.cc:304] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:05.521008 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 (DEFAULT_TABLE table=default.Table_44e92821147643aa9a6b725eadd88b39 [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.521322 5606 raft_consensus.cc:695] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:05.522024 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ac07ac6fc9f477ba7f9accc5533be39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.522199 5606 consensus_queue.cc:237] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.453045 5597 raft_consensus.cc:357] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.525111 5597 raft_consensus.cc:383] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.525153 5689 raft_consensus.cc:2798] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:05.525100 5299 tablet_service.cc:1467] Processing CreateTablet for tablet 2d634add82994f74960cd2b8373a4dc6 (DEFAULT_TABLE table=default.Table_ba448520fdb24cd1be60910f20647848 [id=505b9ca1a68e401aa6f6f28a41f09181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.525353 5597 raft_consensus.cc:738] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.525532 5689 raft_consensus.cc:491] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:05.525836 5689 raft_consensus.cc:3054] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:05.525944 5597 consensus_queue.cc:260] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.526752 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d634add82994f74960cd2b8373a4dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.540210 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 (DEFAULT_TABLE table=default.Table_44e92821147643aa9a6b725eadd88b39 [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.536141 5689 raft_consensus.cc:513] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.542203 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ac07ac6fc9f477ba7f9accc5533be39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.544734 5700 tablet.cc:1721] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 20:57:05.546517 5689 leader_election.cc:290] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:05.547278 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574171c94ca6450daf9fb608dbe9b002" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:05.547768 5176 raft_consensus.cc:3054] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:05.550750 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 2d634add82994f74960cd2b8373a4dc6 (DEFAULT_TABLE table=default.Table_ba448520fdb24cd1be60910f20647848 [id=505b9ca1a68e401aa6f6f28a41f09181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.551735 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d634add82994f74960cd2b8373a4dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.554097 5176 raft_consensus.cc:2463] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:05.555097 5257 leader_election.cc:304] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:05.555910 5689 raft_consensus.cc:2798] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:05.557662 5689 raft_consensus.cc:695] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Becoming Leader. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Running, Role: LEADER
I20250110 20:57:05.558713 5689 consensus_queue.cc:237] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.561851 4897 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_26e52fb26ebe44ef9a2b6e2d51defe22" } new_table_name: "default.Table_5f5319ada8df4fc1b4b3c764b2cbf6d5" modify_external_catalogs: true
I20250110 20:57:05.571938 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ec98243c2d784adfb3ab2a95bd2d89b8" } new_table_name: "default.Table_e412e39463cf4fc0bf8a55ba265ffb25" modify_external_catalogs: true
I20250110 20:57:05.572038 4895 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8df7bbf6722249b2be68514a607459b9" } modify_external_catalogs: true
I20250110 20:57:05.576419 4895 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8df7bbf6722249b2be68514a607459b9" } modify_external_catalogs: true
I20250110 20:57:05.577272 4899 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 637dda10276947d5842ed24bbf044d0d
I20250110 20:57:05.578521 5597 ts_tablet_manager.cc:1428] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.127s user 0.007s sys 0.000s
I20250110 20:57:05.579350 5597 tablet_bootstrap.cc:492] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:05.581789 5299 tablet_service.cc:1514] Processing DeleteTablet for tablet 637dda10276947d5842ed24bbf044d0d with delete_type TABLET_DATA_DELETED (replaced by tablet ba1b57de9e4b4b2ab6c3bce8fe78166a) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.509169 5592 ts_tablet_manager.cc:1918] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.521009 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 (DEFAULT_TABLE table=default.Table_44e92821147643aa9a6b725eadd88b39 [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.528707 5437 tablet_service.cc:1467] Processing CreateTablet for tablet 2d634add82994f74960cd2b8373a4dc6 (DEFAULT_TABLE table=default.Table_ba448520fdb24cd1be60910f20647848 [id=505b9ca1a68e401aa6f6f28a41f09181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:05.531834 5435 tablet_service.cc:1514] Processing DeleteTablet for tablet 637dda10276947d5842ed24bbf044d0d with delete_type TABLET_DATA_DELETED (replaced by tablet ba1b57de9e4b4b2ab6c3bce8fe78166a) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.543428 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "574171c94ca6450daf9fb608dbe9b002" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:05.582758 5592 log.cc:1198] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/332dfe3a0e45468695aeb7d7eb7c30ca
I20250110 20:57:05.583439 5592 ts_tablet_manager.cc:1939] T 332dfe3a0e45468695aeb7d7eb7c30ca P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:05.583846 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 637dda10276947d5842ed24bbf044d0d with delete_type TABLET_DATA_DELETED (replaced by tablet ba1b57de9e4b4b2ab6c3bce8fe78166a) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:05.584651 5255 leader_election.cc:343] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.561511 5691 mvcc.cc:204] Tried to move back new op lower bound from 7112878593535307776 to 7112878591275073536. Current Snapshot: MvccSnapshot[applied={T|T < 7112878593535307776}]
I20250110 20:57:05.588871 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ac07ac6fc9f477ba7f9accc5533be39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.593998 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d634add82994f74960cd2b8373a4dc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:05.599308 4899 catalog_manager.cc:6596] ReplaceTablet: tablet 637dda10276947d5842ed24bbf044d0d deleted and replaced by tablet ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:05.602352 5595 ts_tablet_manager.cc:1918] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.576911 5594 tablet_bootstrap.cc:654] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.579605 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 637dda10276947d5842ed24bbf044d0d with delete_type TABLET_DATA_DELETED (replaced by tablet ba1b57de9e4b4b2ab6c3bce8fe78166a) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.608125 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d0b2c46a454745699731c3406fc3ce90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:05.609508 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0b2c46a454745699731c3406fc3ce90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.610081 5595 log.cc:1198] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/22b5876f954e469f86498a529e5c4a68
I20250110 20:57:05.611200 5595 ts_tablet_manager.cc:1939] T 22b5876f954e469f86498a529e5c4a68 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.615934 5673 raft_consensus.cc:491] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.616356 5673 raft_consensus.cc:513] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.618078 5673 leader_election.cc:290] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:05.620352 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2dc0d3abef5741339fdbb9abeed20b15" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:05.621140 5322 raft_consensus.cc:2388] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in current term 1: Already voted for candidate 46005a5920cc4a3697a5c61b1c271384 in this term.
W20250110 20:57:05.623148 5399 leader_election.cc:343] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: 2dc0d3abef5741339fdbb9abeed20b15
I20250110 20:57:05.623517 5399 leader_election.cc:304] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:05.623826 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 22b5876f954e469f86498a529e5c4a68 (table default.Table_213a26b346b94108a30a80c719397f12 [id=3294cc03112a4958b6a090385afd9790]) successfully deleted
I20250110 20:57:05.624548 5673 raft_consensus.cc:3054] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:05.625072 5595 tablet_replica.cc:331] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:05.625761 5595 raft_consensus.cc:2238] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.626147 5595 raft_consensus.cc:2267] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.626194 5594 tablet_bootstrap.cc:492] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:05.627614 5595 ts_tablet_manager.cc:1905] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.626538 5594 ts_tablet_manager.cc:1397] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.295s user 0.010s sys 0.000s
I20250110 20:57:05.631220 5673 raft_consensus.cc:2743] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:05.633374 5703 tablet.cc:1721] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 20:57:05.637962 4891 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 961fef96e9124aafb1c51e287f603b64
I20250110 20:57:05.639283 4891 catalog_manager.cc:6596] ReplaceTablet: tablet 961fef96e9124aafb1c51e287f603b64 deleted and replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247
I20250110 20:57:05.640898 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 961fef96e9124aafb1c51e287f603b64 with delete_type TABLET_DATA_DELETED (replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.642282 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 961fef96e9124aafb1c51e287f603b64 with delete_type TABLET_DATA_DELETED (replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.654695 5594 raft_consensus.cc:357] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.658735 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 37) took 1372 ms. Trace:
I20250110 20:57:05.659088 5592 rpcz_store.cc:276] 0110 20:57:04.286647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:04.288146 (+ 1499us) service_pool.cc:225] Handling call
0110 20:57:04.956745 (+668599us) ts_tablet_manager.cc:1145] Deleting tablet 332dfe3a0e45468695aeb7d7eb7c30ca
0110 20:57:04.956788 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:05.046054 (+ 89266us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:05.508960 (+462906us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:05.658676 (+149716us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":659847}
I20250110 20:57:05.660523 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 332dfe3a0e45468695aeb7d7eb7c30ca (table default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]) successfully deleted
I20250110 20:57:05.661542 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 38) took 1366 ms. Trace:
I20250110 20:57:05.661856 5592 rpcz_store.cc:276] 0110 20:57:04.295591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:04.296803 (+ 1212us) service_pool.cc:225] Handling call
0110 20:57:05.661180 (+1364377us) ts_tablet_manager.cc:1145] Deleting tablet 5165408eb95e4849886252671dfb18d4
0110 20:57:05.661222 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:05.661488 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1359736}
I20250110 20:57:05.657305 5594 raft_consensus.cc:383] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:05.662925 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 5165408eb95e4849886252671dfb18d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5165408eb95e4849886252671dfb18d4 already in progress: creating tablet
I20250110 20:57:05.663190 5594 raft_consensus.cc:738] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.667227 5600 tablet_bootstrap.cc:492] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:05.667690 5600 ts_tablet_manager.cc:1397] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.177s user 0.011s sys 0.002s
I20250110 20:57:05.670517 5600 raft_consensus.cc:357] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.675730 5592 tablet_replica.cc:331] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:05.676916 5592 raft_consensus.cc:2238] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.677333 5592 raft_consensus.cc:2267] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.679072 5592 ts_tablet_manager.cc:1905] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:05.688865 4897 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 574171c94ca6450daf9fb608dbe9b002 consensus state has no leader
I20250110 20:57:05.671525 5705 mvcc.cc:204] Tried to move back new op lower bound from 7112878593535307776 to 7112878591275073536. Current Snapshot: MvccSnapshot[applied={T|T < 7112878593535307776}]
I20250110 20:57:05.705476 5600 raft_consensus.cc:383] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.709539 5600 raft_consensus.cc:738] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.703012 4897 catalog_manager.cc:5526] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:05.710703 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:05.715507 5710 tablet.cc:1721] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250110 20:57:05.712316 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:05.684720 5595 ts_tablet_manager.cc:1918] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.717927 5595 log.cc:1198] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/29599cfce9ba4875b7aed08f5094ce21
I20250110 20:57:05.718524 5595 ts_tablet_manager.cc:1939] T 29599cfce9ba4875b7aed08f5094ce21 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.719339 5593 ts_tablet_manager.cc:1918] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.719673 5593 log.cc:1198] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/29599cfce9ba4875b7aed08f5094ce21
I20250110 20:57:05.720225 5593 ts_tablet_manager.cc:1939] T 29599cfce9ba4875b7aed08f5094ce21 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:05.719902 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:05.713551 4885 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 04fd9785888141abaab655f0ba74efb9 consensus state has no leader
W20250110 20:57:05.714350 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9155a626d26541ce9d0cf31508f19f15 consensus state has no leader
I20250110 20:57:05.715183 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet e38b659f218d42298cb36752b5ca603d
I20250110 20:57:05.726132 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet e38b659f218d42298cb36752b5ca603d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:05.729326 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet e38b659f218d42298cb36752b5ca603d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.730020 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.730648 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 29599cfce9ba4875b7aed08f5094ce21 (table default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f [id=3bbd1acd709b44899907148df944b20d]) successfully deleted
I20250110 20:57:05.706943 5689 raft_consensus.cc:491] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.711524 5600 consensus_queue.cc:260] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.731518 5689 raft_consensus.cc:513] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:05.733027 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 477b09061a4a4f278720168872a83258 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 477b09061a4a4f278720168872a83258 already in progress: creating tablet
W20250110 20:57:05.734314 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 5165408eb95e4849886252671dfb18d4 because the tablet was not found. No further retry: Not found: Tablet not found: 5165408eb95e4849886252671dfb18d4
I20250110 20:57:05.735450 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:05.736207 5595 tablet_replica.cc:331] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
W20250110 20:57:05.736699 5255 leader_election.cc:343] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.737286 5689 leader_election.cc:290] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:05.737555 5595 raft_consensus.cc:2238] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.663751 5594 consensus_queue.cc:260] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.743593 5600 ts_tablet_manager.cc:1428] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.076s user 0.006s sys 0.000s
I20250110 20:57:05.749903 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:05.750496 5176 raft_consensus.cc:2463] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:05.753388 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet e7bc3aae47f54c618798ff30c9cb75e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:03 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:05.755235 5600 tablet_bootstrap.cc:492] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:05.760886 5593 tablet_replica.cc:331] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:05.761525 5593 raft_consensus.cc:2238] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:05.761821 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.762331 5593 raft_consensus.cc:2267] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.764706 5597 tablet_bootstrap.cc:654] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.768191 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet e38b659f218d42298cb36752b5ca603d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.764755 5594 ts_tablet_manager.cc:1428] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.135s user 0.005s sys 0.000s
I20250110 20:57:05.770912 5594 tablet_bootstrap.cc:492] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:05.770965 5592 ts_tablet_manager.cc:1918] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.771373 5592 log.cc:1198] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/0fd2cc66ab594f849f142d7847a1e0f5
I20250110 20:57:05.772027 5592 ts_tablet_manager.cc:1939] T 0fd2cc66ab594f849f142d7847a1e0f5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:05.773913 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 39) took 1448 ms. Trace:
I20250110 20:57:05.774226 5592 rpcz_store.cc:276] 0110 20:57:04.326178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:04.326325 (+ 147us) service_pool.cc:225] Handling call
0110 20:57:05.675575 (+1349250us) ts_tablet_manager.cc:1145] Deleting tablet 0fd2cc66ab594f849f142d7847a1e0f5
0110 20:57:05.675616 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:05.762676 (+ 87060us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:05.770739 (+ 8063us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:05.773853 (+ 3114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1346091}
I20250110 20:57:05.775319 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 0fd2cc66ab594f849f142d7847a1e0f5 (table default.Table_14adb01640aa40d699cfb94980d7aea3 [id=1c2a6b50e68e4363b5ebf736451866a6]) successfully deleted
I20250110 20:57:05.775591 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 45) took 1007 ms. Trace:
I20250110 20:57:05.775888 5592 rpcz_store.cc:276] 0110 20:57:04.768366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:04.769746 (+ 1380us) service_pool.cc:225] Handling call
0110 20:57:05.775287 (+1005541us) ts_tablet_manager.cc:1145] Deleting tablet b26f58b066f646e5b11ddf20cbbf9e1a
0110 20:57:05.775321 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:05.775540 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1003488}
I20250110 20:57:05.752314 5257 leader_election.cc:304] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
W20250110 20:57:05.777927 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet b26f58b066f646e5b11ddf20cbbf9e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b26f58b066f646e5b11ddf20cbbf9e1a already in progress: creating tablet
I20250110 20:57:05.779632 5689 raft_consensus.cc:2798] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250110 20:57:05.782279 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 29599cfce9ba4875b7aed08f5094ce21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29599cfce9ba4875b7aed08f5094ce21 already in progress: creating tablet
I20250110 20:57:05.782883 5689 raft_consensus.cc:491] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:05.783239 5689 raft_consensus.cc:3054] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:05.784132 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 961fef96e9124aafb1c51e287f603b64 with delete_type TABLET_DATA_DELETED (replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.785161 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 29599cfce9ba4875b7aed08f5094ce21 (table default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f [id=3bbd1acd709b44899907148df944b20d]) successfully deleted
I20250110 20:57:05.795343 5600 tablet_bootstrap.cc:654] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:05.793000 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 22b5876f954e469f86498a529e5c4a68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22b5876f954e469f86498a529e5c4a68 already in progress: creating tablet
I20250110 20:57:05.796015 5612 raft_consensus.cc:491] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.796393 5612 raft_consensus.cc:513] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.798166 5612 leader_election.cc:290] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:05.798480 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 477b09061a4a4f278720168872a83258 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 477b09061a4a4f278720168872a83258 already in progress: creating tablet
I20250110 20:57:05.799940 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9155a626d26541ce9d0cf31508f19f15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:05.744040 5595 raft_consensus.cc:2267] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:05.802314 5257 leader_election.cc:343] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.803434 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9155a626d26541ce9d0cf31508f19f15" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:05.804491 5255 leader_election.cc:343] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.804850 5255 leader_election.cc:304] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:05.805953 5612 raft_consensus.cc:2743] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:05.811174 5593 ts_tablet_manager.cc:1905] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.817359 5595 ts_tablet_manager.cc:1905] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.819317 5597 tablet_bootstrap.cc:492] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:05.819679 5597 ts_tablet_manager.cc:1397] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.240s user 0.011s sys 0.000s
I20250110 20:57:05.821848 5597 raft_consensus.cc:357] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.822440 5597 raft_consensus.cc:383] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.822671 5597 raft_consensus.cc:738] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.823325 5592 tablet_replica.cc:331] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:05.823387 4894 master_service.cc:946] ReplaceTablet: received request to replace tablet 70ac7d62f978433dad004f2908c4bbc4 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:05.823318 5597 consensus_queue.cc:260] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.824620 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_7896d2963d9b4ce78484de7c3e9caf91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:05.825466 5593 ts_tablet_manager.cc:1918] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:05.825985 5592 raft_consensus.cc:2238] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 20:57:05.827406 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7896d2963d9b4ce78484de7c3e9caf91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.829890 5592 raft_consensus.cc:2267] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.831821 5592 ts_tablet_manager.cc:1905] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.832703 5595 ts_tablet_manager.cc:1918] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.833014 5595 log.cc:1198] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/a7e7ffb063594128ba137f9aea86bf15
I20250110 20:57:05.837566 5593 log.cc:1198] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/637dda10276947d5842ed24bbf044d0d
I20250110 20:57:05.838351 5593 ts_tablet_manager.cc:1939] T 637dda10276947d5842ed24bbf044d0d P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:05.842307 5594 tablet_bootstrap.cc:654] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.843130 5595 ts_tablet_manager.cc:1939] T a7e7ffb063594128ba137f9aea86bf15 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.846113 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
I20250110 20:57:05.847131 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 78538b2e7bcc45c7afaa145bad6e5813) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.848480 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_5356ef036bd048488ffadd8d4b4caf58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:05.849462 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 3a520230d9fd4a74914e46f784dcf747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a520230d9fd4a74914e46f784dcf747 already in progress: creating tablet
W20250110 20:57:05.850220 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5356ef036bd048488ffadd8d4b4caf58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.850167 5689 raft_consensus.cc:513] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.852705 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:05.853197 5176 raft_consensus.cc:3054] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:05.853485 5597 ts_tablet_manager.cc:1428] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20250110 20:57:05.860142 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet a7e7ffb063594128ba137f9aea86bf15 (table default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]) successfully deleted
I20250110 20:57:05.861666 5595 tablet_replica.cc:331] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:05.861934 5597 tablet_bootstrap.cc:492] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:05.863052 4893 master_service.cc:946] ReplaceTablet: received request to replace tablet 9155a626d26541ce9d0cf31508f19f15 from {username='slave'} at 127.0.0.1:36568
W20250110 20:57:05.863193 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet e38b659f218d42298cb36752b5ca603d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e38b659f218d42298cb36752b5ca603d already in progress: creating tablet
I20250110 20:57:05.864112 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:05.862668 5595 raft_consensus.cc:2238] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.864950 5595 raft_consensus.cc:2267] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:05.865142 5255 leader_election.cc:343] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.867137 5595 ts_tablet_manager.cc:1905] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.867334 5176 raft_consensus.cc:2463] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:05.868650 5257 leader_election.cc:304] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:05.867794 5612 raft_consensus.cc:491] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.869552 5612 raft_consensus.cc:513] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.870487 5695 raft_consensus.cc:491] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:05.870882 5695 raft_consensus.cc:513] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:05.871773 5612 leader_election.cc:290] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:05.872229 5612 raft_consensus.cc:2798] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:05.872684 5695 leader_election.cc:290] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:05.873500 5689 leader_election.cc:290] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:05.873723 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b26f58b066f646e5b11ddf20cbbf9e1a" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:05.874035 5593 tablet_replica.cc:331] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
W20250110 20:57:05.874256 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 961fef96e9124aafb1c51e287f603b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961fef96e9124aafb1c51e287f603b64 already in progress: creating tablet
I20250110 20:57:05.874643 5593 raft_consensus.cc:2238] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.875037 5593 raft_consensus.cc:2267] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:05.874971 5110 leader_election.cc:343] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.875500 5612 raft_consensus.cc:695] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Becoming Leader. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Running, Role: LEADER
I20250110 20:57:05.876389 5612 consensus_queue.cc:237] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.878757 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ac7d62f978433dad004f2908c4bbc4" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:05.879881 5592 ts_tablet_manager.cc:1918] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.880053 5597 tablet_bootstrap.cc:654] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:05.880210 5592 log.cc:1198] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/2dc0d3abef5741339fdbb9abeed20b15
I20250110 20:57:05.880772 5592 ts_tablet_manager.cc:1939] T 2dc0d3abef5741339fdbb9abeed20b15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
W20250110 20:57:05.879782 5255 leader_election.cc:343] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.882923 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 2dc0d3abef5741339fdbb9abeed20b15 (table default.Table_d2f92229bca74d26baf2508b363c90ad [id=9daaee4eb9574713b7914a551680c52b]) successfully deleted
I20250110 20:57:05.884542 5593 ts_tablet_manager.cc:1905] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:05.887912 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 31b65488c884452d876ced1c3d1efb9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31b65488c884452d876ced1c3d1efb9a already in progress: creating tablet
I20250110 20:57:05.890414 5592 tablet_replica.cc:331] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:05.891562 5592 raft_consensus.cc:2238] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.892055 5592 raft_consensus.cc:2267] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.892238 4899 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_bf82fe64682446fa913015c01a1478fe" } new_table_name: "default.Table_6ecb93f2a2dc4dc88fe950fe54b6cf2d" modify_external_catalogs: true
I20250110 20:57:05.897075 5592 ts_tablet_manager.cc:1905] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.907536 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ac7d62f978433dad004f2908c4bbc4" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:05.910235 5111 leader_election.cc:343] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: b26f58b066f646e5b11ddf20cbbf9e1a
I20250110 20:57:05.910755 5593 ts_tablet_manager.cc:1918] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 20:57:05.910782 5257 leader_election.cc:343] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:05.911087 5593 log.cc:1198] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/b26f58b066f646e5b11ddf20cbbf9e1a
I20250110 20:57:05.911139 5257 leader_election.cc:304] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:05.910612 5111 leader_election.cc:304] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:05.911618 5593 ts_tablet_manager.cc:1939] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:05.912022 5612 raft_consensus.cc:2743] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:05.913044 5111 raft_consensus.cc:2681] T b26f58b066f646e5b11ddf20cbbf9e1a P ac9003be04f340f3886b5625cc928eb0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250110 20:57:05.915745 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet b26f58b066f646e5b11ddf20cbbf9e1a (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]) successfully deleted
I20250110 20:57:05.916759 5595 ts_tablet_manager.cc:1918] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:05.917093 5595 log.cc:1198] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/2dc0d3abef5741339fdbb9abeed20b15
I20250110 20:57:05.917675 5595 ts_tablet_manager.cc:1939] T 2dc0d3abef5741339fdbb9abeed20b15 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.919808 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 2dc0d3abef5741339fdbb9abeed20b15 (table default.Table_d2f92229bca74d26baf2508b363c90ad [id=9daaee4eb9574713b7914a551680c52b]) successfully deleted
I20250110 20:57:05.920279 5595 tablet_replica.cc:331] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
W20250110 20:57:05.920469 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 31b65488c884452d876ced1c3d1efb9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31b65488c884452d876ced1c3d1efb9a already in progress: creating tablet
I20250110 20:57:05.920879 5595 raft_consensus.cc:2238] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.921399 5595 raft_consensus.cc:2267] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:05.927001 5595 ts_tablet_manager.cc:1905] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.928042 4891 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_bb5cbd6c878c4302aa43b01c0f1c3e96" } modify_external_catalogs: true
I20250110 20:57:05.928563 4891 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_bb5cbd6c878c4302aa43b01c0f1c3e96" } modify_external_catalogs: true
I20250110 20:57:05.932817 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_449deb3d40844317af53f2dab86bc610"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:05.934991 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_449deb3d40844317af53f2dab86bc610 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:05.938930 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d6cd02ab71b14c2ab609b021be613df9" } new_table_name: "default.Table_ae86414449b14c9aba13ee321887871a" modify_external_catalogs: true
I20250110 20:57:05.940730 5595 ts_tablet_manager.cc:1918] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250110 20:57:05.940830 4899 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1fafed70adb496d97acfd5fea356292 consensus state has no leader
I20250110 20:57:05.941053 5595 log.cc:1198] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/637dda10276947d5842ed24bbf044d0d
I20250110 20:57:05.941711 5595 ts_tablet_manager.cc:1939] T 637dda10276947d5842ed24bbf044d0d P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:05.944038 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
W20250110 20:57:05.945613 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 961fef96e9124aafb1c51e287f603b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961fef96e9124aafb1c51e287f603b64 already in progress: creating tablet
W20250110 20:57:05.947189 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 3a520230d9fd4a74914e46f784dcf747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a520230d9fd4a74914e46f784dcf747 already in progress: creating tablet
W20250110 20:57:05.948637 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet e38b659f218d42298cb36752b5ca603d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e38b659f218d42298cb36752b5ca603d already in progress: creating tablet
I20250110 20:57:05.939983 5592 ts_tablet_manager.cc:1918] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:05.949314 5592 log.cc:1198] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/637dda10276947d5842ed24bbf044d0d
I20250110 20:57:05.950107 5592 ts_tablet_manager.cc:1939] T 637dda10276947d5842ed24bbf044d0d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:05.952245 5594 tablet_bootstrap.cc:492] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:05.952603 5594 ts_tablet_manager.cc:1397] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.182s user 0.008s sys 0.000s
I20250110 20:57:05.954511 4891 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 6914f4a6ae734de98d48ee6469c3cd1c
I20250110 20:57:05.960688 4899 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_56fa755492734855973131b69735b169" } modify_external_catalogs: true
I20250110 20:57:05.963474 4899 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_56fa755492734855973131b69735b169" } modify_external_catalogs: true
I20250110 20:57:05.966193 5594 raft_consensus.cc:357] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.968456 5597 tablet_bootstrap.cc:492] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:05.968914 5597 ts_tablet_manager.cc:1397] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.107s user 0.013s sys 0.009s
W20250110 20:57:05.968917 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2cbb46ea11c4e7d8bc5993ccdc52247 has no consensus state
I20250110 20:57:05.972615 5597 raft_consensus.cc:357] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.973229 5597 raft_consensus.cc:383] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.973479 5597 raft_consensus.cc:738] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.966813 5594 raft_consensus.cc:383] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:05.974050 5597 consensus_queue.cc:260] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.975919 5597 ts_tablet_manager.cc:1428] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250110 20:57:05.976737 5597 tablet_bootstrap.cc:492] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:05.978695 5600 tablet_bootstrap.cc:492] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:05.979071 5600 ts_tablet_manager.cc:1397] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.224s user 0.008s sys 0.000s
I20250110 20:57:05.981350 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 637dda10276947d5842ed24bbf044d0d (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
W20250110 20:57:05.986768 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 637dda10276947d5842ed24bbf044d0d because the tablet was not found. No further retry: Not found: Tablet not found: 637dda10276947d5842ed24bbf044d0d
W20250110 20:57:05.987802 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 961fef96e9124aafb1c51e287f603b64 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 961fef96e9124aafb1c51e287f603b64 already in progress: creating tablet
W20250110 20:57:05.989711 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet e38b659f218d42298cb36752b5ca603d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e38b659f218d42298cb36752b5ca603d already in progress: creating tablet
I20250110 20:57:05.992656 5594 raft_consensus.cc:738] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:05.993074 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5876f954e469f86498a529e5c4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:05.992671 4892 catalog_manager.cc:5526] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194). New cstate: current_term: 1 leader_uuid: "46005a5920cc4a3697a5c61b1c271384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:05.993717 5593 tablet_replica.cc:331] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:05.993276 5594 consensus_queue.cc:260] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:05.993985 4892 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 574171c94ca6450daf9fb608dbe9b002 (table default.Table_5f5319ada8df4fc1b4b3c764b2cbf6d5 [id=44b872c361cf41a0b7ec0d28e66725ea]). Expected version 1 got 0
I20250110 20:57:05.994323 5593 raft_consensus.cc:2238] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.994707 5593 raft_consensus.cc:2267] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:05.994756 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 3a520230d9fd4a74914e46f784dcf747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a520230d9fd4a74914e46f784dcf747 already in progress: creating tablet
I20250110 20:57:05.995983 5592 tablet_replica.cc:331] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:05.996718 5593 ts_tablet_manager.cc:1905] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:05.996511 5592 raft_consensus.cc:2238] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:05.998230 5592 raft_consensus.cc:2267] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:06.001178 5594 ts_tablet_manager.cc:1428] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20250110 20:57:06.001190 5592 ts_tablet_manager.cc:1905] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:06.009783 5442 tablet_service.cc:1467] Processing CreateTablet for tablet e361fb8b47d74548b5a2326d25a9e28c (DEFAULT_TABLE table=default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.009948 4895 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:06.010846 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e361fb8b47d74548b5a2326d25a9e28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.011271 5150 tablet_service.cc:1467] Processing CreateTablet for tablet e361fb8b47d74548b5a2326d25a9e28c (DEFAULT_TABLE table=default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.012276 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e361fb8b47d74548b5a2326d25a9e28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.016464 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_fc5f72ec93dd40bbaeb1be5da336b098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:06.018143 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc5f72ec93dd40bbaeb1be5da336b098 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.019716 5593 ts_tablet_manager.cc:1918] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:06.020061 5593 log.cc:1198] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/22b5876f954e469f86498a529e5c4a68
I20250110 20:57:06.020627 5593 ts_tablet_manager.cc:1939] T 22b5876f954e469f86498a529e5c4a68 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:06.022804 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 22b5876f954e469f86498a529e5c4a68 (table default.Table_213a26b346b94108a30a80c719397f12 [id=3294cc03112a4958b6a090385afd9790]) successfully deleted
I20250110 20:57:06.022907 4895 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:06.027709 5296 tablet_service.cc:1467] Processing CreateTablet for tablet e361fb8b47d74548b5a2326d25a9e28c (DEFAULT_TABLE table=default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.028769 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e361fb8b47d74548b5a2326d25a9e28c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.037174 5594 tablet_bootstrap.cc:492] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.042579 5594 tablet_bootstrap.cc:654] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.046082 5436 tablet_service.cc:1467] Processing CreateTablet for tablet da24904dece04a6194790054f6d7153a (DEFAULT_TABLE table=default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.047052 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da24904dece04a6194790054f6d7153a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.050482 5299 tablet_service.cc:1467] Processing CreateTablet for tablet da24904dece04a6194790054f6d7153a (DEFAULT_TABLE table=default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.058418 5594 tablet_bootstrap.cc:492] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.058807 5594 ts_tablet_manager.cc:1397] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.000s
I20250110 20:57:06.060019 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da24904dece04a6194790054f6d7153a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.060173 5592 ts_tablet_manager.cc:1918] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:06.060523 5592 log.cc:1198] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/e7bc3aae47f54c618798ff30c9cb75e8
I20250110 20:57:06.061169 5592 ts_tablet_manager.cc:1939] T e7bc3aae47f54c618798ff30c9cb75e8 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:06.060851 5594 raft_consensus.cc:357] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:06.061492 5594 raft_consensus.cc:383] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.061751 5594 raft_consensus.cc:738] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.067293 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet e7bc3aae47f54c618798ff30c9cb75e8 (table default.Table_ccaf097e265d4ed6a3ed2e9c0984229a [id=af9522c35cd74c119bdc1af28f7bab6c]) successfully deleted
I20250110 20:57:06.067339 5594 consensus_queue.cc:260] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:06.076166 4892 catalog_manager.cc:5526] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194). New cstate: current_term: 1 leader_uuid: "46005a5920cc4a3697a5c61b1c271384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
W20250110 20:57:06.079488 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.254:34617, user_credentials={real_user=slave}} blocked reactor thread for 49845.8us
I20250110 20:57:06.083206 5150 tablet_service.cc:1467] Processing CreateTablet for tablet da24904dece04a6194790054f6d7153a (DEFAULT_TABLE table=default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.084247 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da24904dece04a6194790054f6d7153a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.089644 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 2967c06715e241dcbbe784eed2373704 (DEFAULT_TABLE table=default.Table_6d29a7293ace46b186d68a71d0a5a688 [id=4f67b22c80ae49b79ddc245c7e304307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.090723 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2967c06715e241dcbbe784eed2373704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.094504 5594 ts_tablet_manager.cc:1428] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20250110 20:57:06.097127 5594 tablet_bootstrap.cc:492] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.100745 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ba963ec731634f379b0b47ecbc9a1d31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:06.102284 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba963ec731634f379b0b47ecbc9a1d31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.103771 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 2967c06715e241dcbbe784eed2373704 (DEFAULT_TABLE table=default.Table_6d29a7293ace46b186d68a71d0a5a688 [id=4f67b22c80ae49b79ddc245c7e304307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.104770 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2967c06715e241dcbbe784eed2373704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.109342 4894 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 70ac7d62f978433dad004f2908c4bbc4
I20250110 20:57:06.110522 4894 catalog_manager.cc:6596] ReplaceTablet: tablet 70ac7d62f978433dad004f2908c4bbc4 deleted and replaced by tablet 8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:06.110934 5437 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d67e94c8f9a414499b5e20a5cc0ead3) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:06.115046 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 70ac7d62f978433dad004f2908c4bbc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70ac7d62f978433dad004f2908c4bbc4 already in progress: creating tablet
I20250110 20:57:06.118175 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d67e94c8f9a414499b5e20a5cc0ead3) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:06.119738 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 70ac7d62f978433dad004f2908c4bbc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70ac7d62f978433dad004f2908c4bbc4 already in progress: creating tablet
I20250110 20:57:06.132162 5612 raft_consensus.cc:491] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:06.132397 5594 tablet_bootstrap.cc:654] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.132735 5612 raft_consensus.cc:513] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.135077 5612 leader_election.cc:290] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:06.135530 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e43461251413b97721323cd935374" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:06.137377 5255 leader_election.cc:343] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:06.142443 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e43461251413b97721323cd935374" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:06.144255 5257 leader_election.cc:343] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:06.144659 5257 leader_election.cc:304] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:06.145632 5612 raft_consensus.cc:2743] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:06.146384 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d67e94c8f9a414499b5e20a5cc0ead3) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:06.147608 5594 tablet_bootstrap.cc:492] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.147997 5594 ts_tablet_manager.cc:1397] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20250110 20:57:05.981168 5600 raft_consensus.cc:357] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.149003 5595 tablet_replica.cc:331] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:06.149071 5600 raft_consensus.cc:383] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.149367 5600 raft_consensus.cc:738] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.149731 5595 raft_consensus.cc:2238] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:06.150154 5595 raft_consensus.cc:2267] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:06.150174 5594 raft_consensus.cc:357] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.151703 5594 raft_consensus.cc:383] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.151738 5595 ts_tablet_manager.cc:1905] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:06.151937 5594 raft_consensus.cc:738] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.152575 5594 consensus_queue.cc:260] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.157927 5597 tablet_bootstrap.cc:654] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.158492 5594 ts_tablet_manager.cc:1428] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20250110 20:57:06.158690 5600 consensus_queue.cc:260] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.159263 5594 tablet_bootstrap.cc:492] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.166972 4893 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 9155a626d26541ce9d0cf31508f19f15
I20250110 20:57:06.168289 4893 catalog_manager.cc:6596] ReplaceTablet: tablet 9155a626d26541ce9d0cf31508f19f15 deleted and replaced by tablet 41de354807004e26bfb147f1d420037e
I20250110 20:57:06.169462 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 9155a626d26541ce9d0cf31508f19f15 with delete_type TABLET_DATA_DELETED (replaced by tablet 41de354807004e26bfb147f1d420037e) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:06.169507 4891 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f" } modify_external_catalogs: true
I20250110 20:57:06.170019 4891 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f" } modify_external_catalogs: true
I20250110 20:57:06.170709 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_44e92821147643aa9a6b725eadd88b39" } new_table_name: "default.Table_655c267acec64c96a52b2645b21eba95" modify_external_catalogs: true
I20250110 20:57:06.172362 5595 ts_tablet_manager.cc:1918] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 20:57:06.171633 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 9155a626d26541ce9d0cf31508f19f15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9155a626d26541ce9d0cf31508f19f15 already in progress: creating tablet
I20250110 20:57:06.175256 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_64d7746346a94643bdf6debc10a1ac6e" } new_table_name: "default.Table_838d31fde06b486f86d7870e9f580912" modify_external_catalogs: true
I20250110 20:57:06.175822 5595 log.cc:1198] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/70ac7d62f978433dad004f2908c4bbc4
I20250110 20:57:06.175702 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_792d0af8e9d04ad28d950c7656211372"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:06.176436 5595 ts_tablet_manager.cc:1939] T 70ac7d62f978433dad004f2908c4bbc4 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
W20250110 20:57:06.177243 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_792d0af8e9d04ad28d950c7656211372 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.181106 5299 tablet_service.cc:1514] Processing DeleteTablet for tablet 9155a626d26541ce9d0cf31508f19f15 with delete_type TABLET_DATA_DELETED (replaced by tablet 41de354807004e26bfb147f1d420037e) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:06.183712 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 2967c06715e241dcbbe784eed2373704 (DEFAULT_TABLE table=default.Table_6d29a7293ace46b186d68a71d0a5a688 [id=4f67b22c80ae49b79ddc245c7e304307]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.184834 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2967c06715e241dcbbe784eed2373704. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.187040 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_7aff60922b99432480e13cc6cdc61c7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:06.188733 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 70ac7d62f978433dad004f2908c4bbc4 (table default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]) successfully deleted
W20250110 20:57:06.188694 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7aff60922b99432480e13cc6cdc61c7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.189517 5595 tablet_replica.cc:331] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:06.189644 5594 tablet_bootstrap.cc:654] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.190164 5595 raft_consensus.cc:2238] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:06.190551 5595 raft_consensus.cc:2267] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:06.191989 5595 ts_tablet_manager.cc:1905] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:06.193562 4899 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 3786c6b0a3c64647b6a04a0732988a7b
I20250110 20:57:06.209738 5600 ts_tablet_manager.cc:1428] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.230s user 0.007s sys 0.000s
I20250110 20:57:06.214187 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 9155a626d26541ce9d0cf31508f19f15 with delete_type TABLET_DATA_DELETED (replaced by tablet 41de354807004e26bfb147f1d420037e) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:06.219017 5594 tablet_bootstrap.cc:492] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.220955 5594 ts_tablet_manager.cc:1397] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.005s
I20250110 20:57:06.223539 5594 raft_consensus.cc:357] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.224212 5594 raft_consensus.cc:383] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.224448 5594 raft_consensus.cc:738] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
W20250110 20:57:06.226156 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 9155a626d26541ce9d0cf31508f19f15 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9155a626d26541ce9d0cf31508f19f15 already in progress: creating tablet
I20250110 20:57:06.227298 5600 tablet_bootstrap.cc:492] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:06.230358 5594 consensus_queue.cc:260] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.234503 5594 ts_tablet_manager.cc:1428] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.013s user 0.002s sys 0.003s
I20250110 20:57:06.235374 5594 tablet_bootstrap.cc:492] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.235893 5595 ts_tablet_manager.cc:1918] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:06.236248 5595 log.cc:1198] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/9155a626d26541ce9d0cf31508f19f15
I20250110 20:57:06.236832 5595 ts_tablet_manager.cc:1939] T 9155a626d26541ce9d0cf31508f19f15 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:06.246824 5600 tablet_bootstrap.cc:654] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.257714 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 9155a626d26541ce9d0cf31508f19f15 (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]) successfully deleted
I20250110 20:57:06.265080 5594 tablet_bootstrap.cc:654] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.281188 5600 tablet_bootstrap.cc:492] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:06.285364 5600 ts_tablet_manager.cc:1397] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.007s
I20250110 20:57:06.290409 5600 raft_consensus.cc:357] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.295485 5594 tablet_bootstrap.cc:492] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.295910 5594 ts_tablet_manager.cc:1397] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.008s
I20250110 20:57:06.298099 5594 raft_consensus.cc:357] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.298935 5594 raft_consensus.cc:383] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.299181 5594 raft_consensus.cc:738] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
W20250110 20:57:06.299616 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 472174c6885b446a839721157914f047 consensus state has no leader
I20250110 20:57:06.299830 5594 consensus_queue.cc:260] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:06.300748 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ac07ac6fc9f477ba7f9accc5533be39 consensus state has no leader
I20250110 20:57:06.304883 5594 ts_tablet_manager.cc:1428] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250110 20:57:06.305697 5594 tablet_bootstrap.cc:492] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.308961 4891 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet d9776df98cb54be38a521852634490e7
I20250110 20:57:06.312613 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:06.313436 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:06.313992 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet d9776df98cb54be38a521852634490e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9776df98cb54be38a521852634490e7 already in progress: creating tablet
I20250110 20:57:06.314666 5600 raft_consensus.cc:383] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.311111 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:06.315446 5600 raft_consensus.cc:738] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:06.315789 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet d9776df98cb54be38a521852634490e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9776df98cb54be38a521852634490e7 already in progress: creating tablet
I20250110 20:57:06.316628 5600 consensus_queue.cc:260] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:06.317353 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet d9776df98cb54be38a521852634490e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9776df98cb54be38a521852634490e7 already in progress: creating tablet
I20250110 20:57:06.324532 5295 tablet_service.cc:1514] Processing DeleteTablet for tablet 477b09061a4a4f278720168872a83258 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:06.325197 5595 tablet_replica.cc:331] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:06.325918 5595 raft_consensus.cc:2238] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:06.328964 5595 raft_consensus.cc:2267] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:06.329110 5594 tablet_bootstrap.cc:654] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.336683 4892 master_service.cc:946] ReplaceTablet: received request to replace tablet e361fb8b47d74548b5a2326d25a9e28c from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:06.337769 5600 ts_tablet_manager.cc:1428] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.051s user 0.005s sys 0.000s
I20250110 20:57:06.338440 5595 ts_tablet_manager.cc:1905] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:06.338539 5600 tablet_bootstrap.cc:492] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:06.352088 5594 tablet_bootstrap.cc:492] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.352485 5594 ts_tablet_manager.cc:1397] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.004s
I20250110 20:57:06.355048 5600 tablet_bootstrap.cc:654] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.354712 5594 raft_consensus.cc:357] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.355345 5594 raft_consensus.cc:383] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.355589 5594 raft_consensus.cc:738] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.358299 5594 consensus_queue.cc:260] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.368580 5600 tablet_bootstrap.cc:492] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:06.368959 5600 ts_tablet_manager.cc:1397] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.000s
I20250110 20:57:06.370992 5594 ts_tablet_manager.cc:1428] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20250110 20:57:06.371209 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_1ec05f4286b94b13bbe215ff5cba4d32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:06.372853 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ec05f4286b94b13bbe215ff5cba4d32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.373778 5600 raft_consensus.cc:357] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.374389 5600 raft_consensus.cc:383] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.374614 5600 raft_consensus.cc:738] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.375154 5600 consensus_queue.cc:260] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.381559 5594 tablet_bootstrap.cc:492] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.390552 5695 consensus_queue.cc:1035] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:06.391839 4900 master_service.cc:946] ReplaceTablet: received request to replace tablet 31b65488c884452d876ced1c3d1efb9a from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:06.396586 4900 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_7aff60922b99432480e13cc6cdc61c7d" } modify_external_catalogs: true
I20250110 20:57:06.397105 4900 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_7aff60922b99432480e13cc6cdc61c7d" } modify_external_catalogs: true
I20250110 20:57:06.399087 5594 tablet_bootstrap.cc:654] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.400494 5597 tablet_bootstrap.cc:492] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:06.400878 5597 ts_tablet_manager.cc:1397] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.424s user 0.008s sys 0.000s
I20250110 20:57:06.402968 5597 raft_consensus.cc:357] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.403559 5597 raft_consensus.cc:383] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.403802 5597 raft_consensus.cc:738] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.412266 5594 tablet_bootstrap.cc:492] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.412647 5594 ts_tablet_manager.cc:1397] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.003s
I20250110 20:57:06.414842 5594 raft_consensus.cc:357] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.415508 5594 raft_consensus.cc:383] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.415752 5594 raft_consensus.cc:738] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.416302 5594 consensus_queue.cc:260] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.425434 4900 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 285a763445734e1d90b7d93fa570d094
I20250110 20:57:06.428371 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a3a9cc7de02e4fba887c5bc79ffd6713"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:06.430002 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.434655 5600 ts_tablet_manager.cc:1428] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.065s user 0.003s sys 0.003s
I20250110 20:57:06.435439 5600 tablet_bootstrap.cc:492] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:06.436288 5176 raft_consensus.cc:1270] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Refusing update from remote peer 46005a5920cc4a3697a5c61b1c271384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:06.437878 5594 ts_tablet_manager.cc:1428] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20250110 20:57:06.443724 5463 raft_consensus.cc:3054] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:06.451869 5463 raft_consensus.cc:1270] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Refusing update from remote peer 46005a5920cc4a3697a5c61b1c271384: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:06.451607 5597 consensus_queue.cc:260] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.453372 5713 consensus_queue.cc:1035] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:06.456053 5612 consensus_queue.cc:1035] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:06.465340 5597 ts_tablet_manager.cc:1428] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.064s user 0.004s sys 0.002s
I20250110 20:57:06.466207 5597 tablet_bootstrap.cc:492] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:06.466756 5594 tablet_bootstrap.cc:492] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.503002 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_449deb3d40844317af53f2dab86bc610" } new_table_name: "default.Table_5053ea7526a74f6da2ece254f837bbae" modify_external_catalogs: true
I20250110 20:57:06.507010 5600 tablet_bootstrap.cc:654] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.511574 5595 ts_tablet_manager.cc:1918] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:06.511926 5595 log.cc:1198] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/477b09061a4a4f278720168872a83258
I20250110 20:57:06.514998 5597 tablet_bootstrap.cc:654] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.516273 5736 tablet.cc:1721] T 574171c94ca6450daf9fb608dbe9b002 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:06.517511 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_655c267acec64c96a52b2645b21eba95" } new_table_name: "default.Table_057bb914f63b44f28527df30e38ce58f" modify_external_catalogs: true
I20250110 20:57:06.517665 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f4a1fe05f073450782146ba7b185060b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:06.519073 5594 tablet_bootstrap.cc:654] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:06.519156 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4a1fe05f073450782146ba7b185060b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.520607 4893 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_fc5f72ec93dd40bbaeb1be5da336b098" } modify_external_catalogs: true
I20250110 20:57:06.511854 5735 tablet.cc:1721] T 574171c94ca6450daf9fb608dbe9b002 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:06.521111 4893 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_fc5f72ec93dd40bbaeb1be5da336b098" } modify_external_catalogs: true
I20250110 20:57:06.526153 5595 ts_tablet_manager.cc:1939] T 477b09061a4a4f278720168872a83258 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:06.530210 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 477b09061a4a4f278720168872a83258 (table default.Table_655c267acec64c96a52b2645b21eba95 [id=c40af9e5c0e5472580ecfe5423b124e5]) successfully deleted
I20250110 20:57:06.531566 4896 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 574171c94ca6450daf9fb608dbe9b002 (table default.Table_5f5319ada8df4fc1b4b3c764b2cbf6d5 [id=44b872c361cf41a0b7ec0d28e66725ea]). Expected version 1 got 0
I20250110 20:57:06.532577 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 5165408eb95e4849886252671dfb18d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:06.533259 5593 tablet_replica.cc:331] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:06.534041 5593 raft_consensus.cc:2238] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:06.534442 5593 raft_consensus.cc:2267] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:06.536897 5593 ts_tablet_manager.cc:1905] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:06.537379 5463 raft_consensus.cc:3054] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:06.547127 5600 tablet_bootstrap.cc:492] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:06.547487 5600 ts_tablet_manager.cc:1397] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.112s user 0.006s sys 0.001s
I20250110 20:57:06.550837 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a1f70418cc2f433d90434c4621fd49d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:06.558013 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1f70418cc2f433d90434c4621fd49d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.566241 5731 mvcc.cc:204] Tried to move back new op lower bound from 7112878597817933824 to 7112878594304114688. Current Snapshot: MvccSnapshot[applied={T|T < 7112878597817933824}]
I20250110 20:57:06.567169 5673 raft_consensus.cc:491] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:06.567591 5673 raft_consensus.cc:513] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.569193 5593 ts_tablet_manager.cc:1918] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:06.569527 5593 log.cc:1198] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/5165408eb95e4849886252671dfb18d4
I20250110 20:57:06.570143 5593 ts_tablet_manager.cc:1939] T 5165408eb95e4849886252671dfb18d4 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:06.569240 5673 leader_election.cc:290] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:06.570318 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472174c6885b446a839721157914f047" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:06.571890 5399 leader_election.cc:343] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:06.572234 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472174c6885b446a839721157914f047" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
W20250110 20:57:06.573565 5398 leader_election.cc:343] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:06.573971 5398 leader_election.cc:304] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:06.573385 5743 tablet.cc:1721] T 574171c94ca6450daf9fb608dbe9b002 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:06.575045 5673 raft_consensus.cc:2743] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:06.578845 5597 tablet_bootstrap.cc:492] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:06.579200 5597 ts_tablet_manager.cc:1397] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.113s user 0.006s sys 0.002s
I20250110 20:57:06.582499 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 5165408eb95e4849886252671dfb18d4 (table default.Table_1367656b1bef487abdee0e294f835b5f [id=ef270c5c9b0d465b8333565e19edd86b]) successfully deleted
W20250110 20:57:06.585986 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d310cf52aa8c4930a7821117a46dc958 has no consensus state
W20250110 20:57:06.587659 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ac07ac6fc9f477ba7f9accc5533be39 consensus state has no leader
I20250110 20:57:06.596213 5600 raft_consensus.cc:357] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:06.599380 5600 raft_consensus.cc:383] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.599337 5739 consensus_queue.cc:1035] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:06.600010 5600 raft_consensus.cc:738] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.602533 5673 raft_consensus.cc:491] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:06.603123 5673 raft_consensus.cc:513] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:06.605268 5673 leader_election.cc:290] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:06.607367 5594 tablet_bootstrap.cc:492] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.604390 5600 consensus_queue.cc:260] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:06.607724 5594 ts_tablet_manager.cc:1397] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.003s
I20250110 20:57:06.608685 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:06.608593 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:06.611459 5594 raft_consensus.cc:357] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.612075 5594 raft_consensus.cc:383] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.612319 5594 raft_consensus.cc:738] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.612862 5594 consensus_queue.cc:260] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.609265 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 574171c94ca6450daf9fb608dbe9b002 (table default.Table_5f5319ada8df4fc1b4b3c764b2cbf6d5 [id=44b872c361cf41a0b7ec0d28e66725ea]). Expected version 1 got 0
I20250110 20:57:06.614763 5594 ts_tablet_manager.cc:1428] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250110 20:57:06.616034 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_b89da7c5e86341c39853bbacce851ad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:06.616597 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01962c5b7ae64f80b998948d18250b4c" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:06.617262 5176 raft_consensus.cc:2388] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in current term 1: Already voted for candidate 46005a5920cc4a3697a5c61b1c271384 in this term.
W20250110 20:57:06.617734 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b89da7c5e86341c39853bbacce851ad4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.618120 5399 leader_election.cc:304] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:06.624086 5600 ts_tablet_manager.cc:1428] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.076s user 0.006s sys 0.001s
I20250110 20:57:06.628855 5600 tablet_bootstrap.cc:492] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:06.629398 4894 master_service.cc:946] ReplaceTablet: received request to replace tablet 961fef96e9124aafb1c51e287f603b64 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:06.632156 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d5cee3d339fd4e289d63ec604aface2e" } modify_external_catalogs: true
I20250110 20:57:06.632625 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d5cee3d339fd4e289d63ec604aface2e" } modify_external_catalogs: true
I20250110 20:57:06.640385 5746 raft_consensus.cc:3054] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:06.659616 5597 raft_consensus.cc:357] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.660306 5597 raft_consensus.cc:383] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.660544 5597 raft_consensus.cc:738] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.661098 5597 consensus_queue.cc:260] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.670053 5594 tablet_bootstrap.cc:492] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.671329 5748 raft_consensus.cc:491] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:06.680445 4895 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 574171c94ca6450daf9fb608dbe9b002 (table default.Table_5f5319ada8df4fc1b4b3c764b2cbf6d5 [id=44b872c361cf41a0b7ec0d28e66725ea]). Expected version 1 got 0
I20250110 20:57:06.672979 5748 raft_consensus.cc:513] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.686043 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b5876f954e469f86498a529e5c4a68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:06.686695 5592 tablet_replica.cc:331] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:06.686800 5748 leader_election.cc:290] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:06.687347 5592 raft_consensus.cc:2238] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:06.687728 5592 raft_consensus.cc:2267] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:06.688593 5398 leader_election.cc:343] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 29599cfce9ba4875b7aed08f5094ce21
I20250110 20:57:06.692798 5597 ts_tablet_manager.cc:1428] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.095s user 0.005s sys 0.000s
I20250110 20:57:06.695079 4893 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:06.698113 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:06.699541 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:06.700448 5597 tablet_bootstrap.cc:492] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:06.701797 5592 ts_tablet_manager.cc:1905] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:06.702021 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 7cf0c87159574bfcbc7a34abe508f9d6 because the tablet was not found. No further retry: Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:06.706681 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:06.708792 5594 tablet_bootstrap.cc:654] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.710640 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet e38b659f218d42298cb36752b5ca603d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:06.711393 5593 tablet_replica.cc:331] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:06.712059 5593 raft_consensus.cc:2238] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:06.712477 5593 raft_consensus.cc:2267] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:06.713567 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:06.714576 5593 ts_tablet_manager.cc:1905] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:06.721446 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:06.730191 5739 raft_consensus.cc:491] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:06.730604 5739 raft_consensus.cc:513] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:06.732367 5739 leader_election.cc:290] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:06.733750 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5256d081843a492d8081c4319f3446d5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:06.734622 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5256d081843a492d8081c4319f3446d5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
W20250110 20:57:06.734944 5110 leader_election.cc:343] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:06.735263 5323 raft_consensus.cc:2463] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:06.728926 4892 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet e361fb8b47d74548b5a2326d25a9e28c
I20250110 20:57:06.737100 5594 tablet_bootstrap.cc:492] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.737481 5594 ts_tablet_manager.cc:1397] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.003s
I20250110 20:57:06.737829 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:06.738822 5111 leader_election.cc:304] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:06.739086 4892 catalog_manager.cc:6596] ReplaceTablet: tablet e361fb8b47d74548b5a2326d25a9e28c deleted and replaced by tablet 491626848ad54be3a18186c65d6dbb60
I20250110 20:57:06.739879 5739 raft_consensus.cc:2798] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:06.739612 5594 raft_consensus.cc:357] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.740181 5739 raft_consensus.cc:491] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:06.740211 5594 raft_consensus.cc:383] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.740447 5594 raft_consensus.cc:738] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.740459 5739 raft_consensus.cc:3054] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:06.740990 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:06.741001 5594 consensus_queue.cc:260] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.729743 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:06.743436 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:06.744076 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:06.745041 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:06.745846 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet e361fb8b47d74548b5a2326d25a9e28c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e361fb8b47d74548b5a2326d25a9e28c already in progress: creating tablet
W20250110 20:57:06.746556 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a because the tablet was not found. No further retry: Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:06.748463 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:06.750804 5399 leader_election.cc:343] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: 29599cfce9ba4875b7aed08f5094ce21
I20250110 20:57:06.755230 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:06.755899 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]). Expected version 1 got 0
I20250110 20:57:06.755177 5746 raft_consensus.cc:2743] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:06.758229 5082 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:06.757135 5399 leader_election.cc:304] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:06.759255 5597 tablet_bootstrap.cc:654] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.763278 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:06.763548 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:06.764855 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a because the tablet was not found. No further retry: Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:06.765604 5600 tablet_bootstrap.cc:654] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.767681 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:06.770166 5594 ts_tablet_manager.cc:1428] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250110 20:57:06.771018 5594 tablet_bootstrap.cc:492] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.773598 5746 raft_consensus.cc:2743] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:06.776688 5296 tablet_service.cc:1467] Processing CreateTablet for tablet d310cf52aa8c4930a7821117a46dc958 (DEFAULT_TABLE table=default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.780893 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d310cf52aa8c4930a7821117a46dc958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.790091 5594 tablet_bootstrap.cc:654] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:06.791194 4886 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d310cf52aa8c4930a7821117a46dc958 consensus state has no leader
I20250110 20:57:06.799850 4896 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ba963ec731634f379b0b47ecbc9a1d31" } new_table_name: "default.Table_5d22ef7f33a3452db4b509894cf54587" modify_external_catalogs: true
I20250110 20:57:06.800705 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_c3b44e7a68ac47a1ad0f7c5496130418"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:06.802299 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f8573af69b094101bbd0f64ce7c1716c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:06.805603 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8573af69b094101bbd0f64ce7c1716c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:06.802729 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3b44e7a68ac47a1ad0f7c5496130418 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:06.812793 4892 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_f4a1fe05f073450782146ba7b185060b" } new_table_name: "default.Table_3cccb8b0fa3f403391692f5f3c691637" modify_external_catalogs: true
I20250110 20:57:06.834410 5713 raft_consensus.cc:491] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:06.834905 5713 raft_consensus.cc:513] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.836691 5713 leader_election.cc:290] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:06.838186 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd9785888141abaab655f0ba74efb9" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:06.838994 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd9785888141abaab655f0ba74efb9" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:06.839538 5255 leader_election.cc:343] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:06.840317 5257 leader_election.cc:343] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:06.840672 5257 leader_election.cc:304] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:06.841877 5436 tablet_service.cc:1467] Processing CreateTablet for tablet d310cf52aa8c4930a7821117a46dc958 (DEFAULT_TABLE table=default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.843479 5713 raft_consensus.cc:2743] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:06.846148 5299 tablet_service.cc:1467] Processing CreateTablet for tablet f0b5ba9f38c1493bba01180744f080e1 (DEFAULT_TABLE table=default.Table_5356ef036bd048488ffadd8d4b4caf58 [id=cdc76f7bfc594bb8bc9f56ccca1a2155]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.847157 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b5ba9f38c1493bba01180744f080e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.847441 5442 tablet_service.cc:1467] Processing CreateTablet for tablet f0b5ba9f38c1493bba01180744f080e1 (DEFAULT_TABLE table=default.Table_5356ef036bd048488ffadd8d4b4caf58 [id=cdc76f7bfc594bb8bc9f56ccca1a2155]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.849531 5150 tablet_service.cc:1467] Processing CreateTablet for tablet f0b5ba9f38c1493bba01180744f080e1 (DEFAULT_TABLE table=default.Table_5356ef036bd048488ffadd8d4b4caf58 [id=cdc76f7bfc594bb8bc9f56ccca1a2155]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.850495 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b5ba9f38c1493bba01180744f080e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.854188 5592 ts_tablet_manager.cc:1918] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:06.854535 5592 log.cc:1198] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/22b5876f954e469f86498a529e5c4a68
I20250110 20:57:06.855144 5592 ts_tablet_manager.cc:1939] T 22b5876f954e469f86498a529e5c4a68 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:06.845187 5153 tablet_service.cc:1467] Processing CreateTablet for tablet d310cf52aa8c4930a7821117a46dc958 (DEFAULT_TABLE table=default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.855911 5597 tablet_bootstrap.cc:492] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:06.855783 5739 raft_consensus.cc:513] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:06.856415 5597 ts_tablet_manager.cc:1397] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.156s user 0.009s sys 0.000s
I20250110 20:57:06.857450 5739 leader_election.cc:290] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:06.858487 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5256d081843a492d8081c4319f3446d5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:06.859624 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0b5ba9f38c1493bba01180744f080e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:06.859850 5110 leader_election.cc:343] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:06.860761 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d310cf52aa8c4930a7821117a46dc958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.862965 5594 tablet_bootstrap.cc:492] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.856221 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d310cf52aa8c4930a7821117a46dc958. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.863327 5594 ts_tablet_manager.cc:1397] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.092s user 0.016s sys 0.008s
I20250110 20:57:06.864672 5295 tablet_service.cc:1467] Processing CreateTablet for tablet d6176c29f01b4ca5888c844c131270f2 (DEFAULT_TABLE table=default.Table_7896d2963d9b4ce78484de7c3e9caf91 [id=a4f704f781a244038555bc9f0a7ad03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.865695 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6176c29f01b4ca5888c844c131270f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.865552 5594 raft_consensus.cc:357] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.865725 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5256d081843a492d8081c4319f3446d5" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:06.866168 5594 raft_consensus.cc:383] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.865914 5742 consensus_queue.cc:1035] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:06.866307 5323 raft_consensus.cc:3054] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:06.866397 5594 raft_consensus.cc:738] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.873636 5594 consensus_queue.cc:260] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.879081 5594 ts_tablet_manager.cc:1428] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250110 20:57:06.882648 5600 tablet_bootstrap.cc:492] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:06.883522 5594 tablet_bootstrap.cc:492] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.896350 5594 tablet_bootstrap.cc:654] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.896868 5437 tablet_service.cc:1467] Processing CreateTablet for tablet d6176c29f01b4ca5888c844c131270f2 (DEFAULT_TABLE table=default.Table_7896d2963d9b4ce78484de7c3e9caf91 [id=a4f704f781a244038555bc9f0a7ad03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.898058 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6176c29f01b4ca5888c844c131270f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.920111 5594 tablet_bootstrap.cc:492] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:06.920506 5594 ts_tablet_manager.cc:1397] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20250110 20:57:06.922606 5594 raft_consensus.cc:357] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.923249 5594 raft_consensus.cc:383] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.923485 5594 raft_consensus.cc:738] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:06.923990 5594 consensus_queue.cc:260] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.933882 5594 ts_tablet_manager.cc:1428] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20250110 20:57:06.935223 5600 ts_tablet_manager.cc:1397] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.307s user 0.011s sys 0.009s
I20250110 20:57:06.935007 5594 tablet_bootstrap.cc:492] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:06.951009 5593 ts_tablet_manager.cc:1918] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:06.951355 5593 log.cc:1198] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/e38b659f218d42298cb36752b5ca603d
I20250110 20:57:06.951977 5593 ts_tablet_manager.cc:1939] T e38b659f218d42298cb36752b5ca603d P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:06.952605 5323 raft_consensus.cc:2463] T 5256d081843a492d8081c4319f3446d5 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:06.953768 5111 leader_election.cc:304] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:06.954706 5739 raft_consensus.cc:2798] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:06.955106 5739 raft_consensus.cc:695] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:06.956005 5594 tablet_bootstrap.cc:654] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:06.956171 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 22b5876f954e469f86498a529e5c4a68 (table default.Table_213a26b346b94108a30a80c719397f12 [id=3294cc03112a4958b6a090385afd9790]) successfully deleted
I20250110 20:57:06.955821 5739 consensus_queue.cc:237] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:06.956779 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 7cf0c87159574bfcbc7a34abe508f9d6 because the tablet was not found. No further retry: Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
W20250110 20:57:06.957183 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet e361fb8b47d74548b5a2326d25a9e28c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e361fb8b47d74548b5a2326d25a9e28c already in progress: creating tablet
W20250110 20:57:06.960327 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a because the tablet was not found. No further retry: Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
W20250110 20:57:06.960857 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a because the tablet was not found. No further retry: Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:06.987306 5746 raft_consensus.cc:491] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:06.987330 5150 tablet_service.cc:1467] Processing CreateTablet for tablet d6176c29f01b4ca5888c844c131270f2 (DEFAULT_TABLE table=default.Table_7896d2963d9b4ce78484de7c3e9caf91 [id=a4f704f781a244038555bc9f0a7ad03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.987784 5746 raft_consensus.cc:513] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.988355 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6176c29f01b4ca5888c844c131270f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:06.989713 5746 leader_election.cc:290] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:06.990335 5600 raft_consensus.cc:357] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.991075 5600 raft_consensus.cc:383] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:06.991326 5600 raft_consensus.cc:738] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:06.992172 5398 leader_election.cc:343] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: b26f58b066f646e5b11ddf20cbbf9e1a
I20250110 20:57:06.991901 5600 consensus_queue.cc:260] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:06.992866 5612 consensus_queue.cc:1035] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:06.997053 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 (DEFAULT_TABLE table=default.Table_fc5f72ec93dd40bbaeb1be5da336b098 [id=6cc26e56afd04d78abff401b7ae32d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:06.999708 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:07.002336 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_1ae0e1ebb3fa451baeb54dfb50eef3a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:07.004451 4899 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5356ef036bd048488ffadd8d4b4caf58" } modify_external_catalogs: true
I20250110 20:57:07.005048 4899 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5356ef036bd048488ffadd8d4b4caf58" } modify_external_catalogs: true
I20250110 20:57:07.010637 4898 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_6d29a7293ace46b186d68a71d0a5a688" } modify_external_catalogs: true
I20250110 20:57:07.011217 4898 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_6d29a7293ace46b186d68a71d0a5a688" } modify_external_catalogs: true
W20250110 20:57:07.014277 4892 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da6a9a054aa54455b4979e4c633ad6b4 has no consensus state
I20250110 20:57:07.015314 4892 master_service.cc:946] ReplaceTablet: received request to replace tablet da24904dece04a6194790054f6d7153a from {username='slave'} at 127.0.0.1:36568
W20250110 20:57:07.016598 5399 leader_election.cc:343] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: b26f58b066f646e5b11ddf20cbbf9e1a
I20250110 20:57:07.017122 5399 leader_election.cc:304] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:07.018520 5746 raft_consensus.cc:2743] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:07.020671 4896 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fde470279464496aa7efdee3800754f3 has no consensus state
W20250110 20:57:07.021996 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ae0e1ebb3fa451baeb54dfb50eef3a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:07.023700 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet e38b659f218d42298cb36752b5ca603d (table default.Table_4a2a0aff7e744f57bc2c98b76bb0a199 [id=ba089f4f57284f80ac3621a88610fec1]) successfully deleted
W20250110 20:57:07.024264 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 7cf0c87159574bfcbc7a34abe508f9d6 because the tablet was not found. No further retry: Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
W20250110 20:57:07.024777 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a because the tablet was not found. No further retry: Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
W20250110 20:57:07.025314 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet e361fb8b47d74548b5a2326d25a9e28c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e361fb8b47d74548b5a2326d25a9e28c already in progress: creating tablet
W20250110 20:57:07.025883 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a because the tablet was not found. No further retry: Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
W20250110 20:57:07.001288 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:07.033880 4895 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_5f5319ada8df4fc1b4b3c764b2cbf6d5 [id=44b872c361cf41a0b7ec0d28e66725ea] alter complete (version 1)
I20250110 20:57:07.035254 5597 raft_consensus.cc:357] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.035912 5597 raft_consensus.cc:383] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.036180 5597 raft_consensus.cc:738] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.039340 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 5165408eb95e4849886252671dfb18d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:04 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.039999 5592 tablet_replica.cc:331] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:07.040621 5592 raft_consensus.cc:2238] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:07.041028 5592 raft_consensus.cc:2267] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:07.042876 5592 ts_tablet_manager.cc:1905] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:07.043481 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 29599cfce9ba4875b7aed08f5094ce21 with delete_type TABLET_DATA_DELETED (replaced by tablet d9776df98cb54be38a521852634490e7) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.044060 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet b26f58b066f646e5b11ddf20cbbf9e1a with delete_type TABLET_DATA_DELETED (replaced by tablet 78538b2e7bcc45c7afaa145bad6e5813) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.064690 4897 catalog_manager.cc:5526] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:07.073904 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet 5793d909c6364047abc85ea5f993c521 from {username='slave'} at 127.0.0.1:36568
W20250110 20:57:07.076550 4887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250110 20:57:07.062203 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet e38b659f218d42298cb36752b5ca603d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:07.082996 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet e38b659f218d42298cb36752b5ca603d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e38b659f218d42298cb36752b5ca603d already in progress: creating tablet
I20250110 20:57:07.036748 5597 consensus_queue.cc:260] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.084818 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf0c87159574bfcbc7a34abe508f9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.090922 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 (DEFAULT_TABLE table=default.Table_fc5f72ec93dd40bbaeb1be5da336b098 [id=6cc26e56afd04d78abff401b7ae32d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.092711 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 7462d21b91c2441fb9e11383466e9ca7 (DEFAULT_TABLE table=default.Table_d0b2c46a454745699731c3406fc3ce90 [id=12aaff501d444161b23ca0484538a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.093111 5597 ts_tablet_manager.cc:1428] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.236s user 0.002s sys 0.004s
I20250110 20:57:07.093899 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7462d21b91c2441fb9e11383466e9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.094573 5597 tablet_bootstrap.cc:492] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:07.100024 5746 raft_consensus.cc:491] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.100448 5746 raft_consensus.cc:513] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.102195 5746 leader_election.cc:290] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:07.103497 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9155a626d26541ce9d0cf31508f19f15" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:07.104660 5399 leader_election.cc:343] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:07.108135 5398 leader_election.cc:343] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 9155a626d26541ce9d0cf31508f19f15
I20250110 20:57:07.108500 5398 leader_election.cc:304] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:07.110783 5597 tablet_bootstrap.cc:654] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.112339 5746 raft_consensus.cc:2743] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:07.117100 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 (DEFAULT_TABLE table=default.Table_fc5f72ec93dd40bbaeb1be5da336b098 [id=6cc26e56afd04d78abff401b7ae32d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.118168 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf0c87159574bfcbc7a34abe508f9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.122321 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 9155a626d26541ce9d0cf31508f19f15 with delete_type TABLET_DATA_DELETED (replaced by tablet 41de354807004e26bfb147f1d420037e) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.132908 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 7462d21b91c2441fb9e11383466e9ca7 (DEFAULT_TABLE table=default.Table_d0b2c46a454745699731c3406fc3ce90 [id=12aaff501d444161b23ca0484538a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.133181 5594 tablet_bootstrap.cc:492] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:07.133541 5594 ts_tablet_manager.cc:1397] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.199s user 0.038s sys 0.042s
I20250110 20:57:07.135676 5594 raft_consensus.cc:357] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.136340 5594 raft_consensus.cc:383] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.136590 5594 raft_consensus.cc:738] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.137128 5594 consensus_queue.cc:260] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.140479 5299 tablet_service.cc:1467] Processing CreateTablet for tablet a7a09ea54e79485b9892d52d0e04da85 (DEFAULT_TABLE table=default.Table_bb34f3dffd1d417690819599206a7a92 [id=0f7debac7d1e4de694325d88c01a5f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.140861 5594 ts_tablet_manager.cc:1428] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:57:07.141696 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a09ea54e79485b9892d52d0e04da85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.141937 5594 tablet_bootstrap.cc:492] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:07.147689 5594 tablet_bootstrap.cc:654] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.150132 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf0c87159574bfcbc7a34abe508f9d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.150803 5597 tablet_bootstrap.cc:492] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:07.151208 5597 ts_tablet_manager.cc:1397] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.057s user 0.009s sys 0.002s
I20250110 20:57:07.151104 5713 raft_consensus.cc:491] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.151854 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7462d21b91c2441fb9e11383466e9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.151477 5713 raft_consensus.cc:513] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.153306 5597 raft_consensus.cc:357] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.153930 5597 raft_consensus.cc:383] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.154166 5597 raft_consensus.cc:738] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.154672 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5793d909c6364047abc85ea5f993c521" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:07.153147 5713 leader_election.cc:290] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:07.156242 5257 leader_election.cc:343] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:07.155649 5597 consensus_queue.cc:260] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.160987 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5793d909c6364047abc85ea5f993c521" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:07.162405 5597 ts_tablet_manager.cc:1428] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250110 20:57:07.163623 5597 tablet_bootstrap.cc:492] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:07.166332 5255 leader_election.cc:343] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:07.166716 5255 leader_election.cc:304] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:07.169852 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 7462d21b91c2441fb9e11383466e9ca7 (DEFAULT_TABLE table=default.Table_d0b2c46a454745699731c3406fc3ce90 [id=12aaff501d444161b23ca0484538a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.170877 5713 raft_consensus.cc:2743] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:07.171871 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d67e94c8f9a414499b5e20a5cc0ead3) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.172960 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7462d21b91c2441fb9e11383466e9ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.178970 5437 tablet_service.cc:1514] Processing DeleteTablet for tablet e38b659f218d42298cb36752b5ca603d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.189950 5597 tablet_bootstrap.cc:654] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.196652 5600 ts_tablet_manager.cc:1428] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.260s user 0.005s sys 0.000s
I20250110 20:57:07.218523 5437 tablet_service.cc:1467] Processing CreateTablet for tablet a7a09ea54e79485b9892d52d0e04da85 (DEFAULT_TABLE table=default.Table_bb34f3dffd1d417690819599206a7a92 [id=0f7debac7d1e4de694325d88c01a5f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.244524 5150 tablet_service.cc:1467] Processing CreateTablet for tablet a7a09ea54e79485b9892d52d0e04da85 (DEFAULT_TABLE table=default.Table_bb34f3dffd1d417690819599206a7a92 [id=0f7debac7d1e4de694325d88c01a5f76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.245712 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a09ea54e79485b9892d52d0e04da85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.251768 5592 ts_tablet_manager.cc:1918] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:07.252151 5592 log.cc:1198] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/5165408eb95e4849886252671dfb18d4
I20250110 20:57:07.252769 5592 ts_tablet_manager.cc:1939] T 5165408eb95e4849886252671dfb18d4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:07.252687 5295 tablet_service.cc:1467] Processing CreateTablet for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (DEFAULT_TABLE table=default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.253921 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2cbb46ea11c4e7d8bc5993ccdc52247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.258491 5597 tablet_bootstrap.cc:492] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:07.258899 5597 ts_tablet_manager.cc:1397] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.095s user 0.012s sys 0.011s
I20250110 20:57:07.259264 5600 tablet_bootstrap.cc:492] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.261054 5597 raft_consensus.cc:357] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.261268 5153 tablet_service.cc:1467] Processing CreateTablet for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (DEFAULT_TABLE table=default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.261695 5597 raft_consensus.cc:383] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.261957 5597 raft_consensus.cc:738] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.262212 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2cbb46ea11c4e7d8bc5993ccdc52247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.262619 5592 tablet_replica.cc:331] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:07.265729 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 5165408eb95e4849886252671dfb18d4 (table default.Table_1367656b1bef487abdee0e294f835b5f [id=ef270c5c9b0d465b8333565e19edd86b]) successfully deleted
I20250110 20:57:07.268069 5592 raft_consensus.cc:2238] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:07.268477 5592 raft_consensus.cc:2267] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:07.269186 5597 consensus_queue.cc:260] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.270263 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_35bec2f295574ca1bf89eaea0cd3942c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:07.270802 5592 ts_tablet_manager.cc:1905] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:07.271850 5597 ts_tablet_manager.cc:1428] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
W20250110 20:57:07.271876 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35bec2f295574ca1bf89eaea0cd3942c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:07.274034 5597 tablet_bootstrap.cc:492] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:07.279412 5597 tablet_bootstrap.cc:654] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.280727 4892 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet da24904dece04a6194790054f6d7153a
I20250110 20:57:07.282302 4892 catalog_manager.cc:6596] ReplaceTablet: tablet da24904dece04a6194790054f6d7153a deleted and replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e
I20250110 20:57:07.286309 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet da24904dece04a6194790054f6d7153a with delete_type TABLET_DATA_DELETED (replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:07.287950 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet da24904dece04a6194790054f6d7153a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da24904dece04a6194790054f6d7153a already in progress: creating tablet
I20250110 20:57:07.290223 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet e38b659f218d42298cb36752b5ca603d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:07.290911 5595 tablet_replica.cc:331] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:07.291558 5595 raft_consensus.cc:2238] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:07.291582 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_6ecb93f2a2dc4dc88fe950fe54b6cf2d" } new_table_name: "default.Table_5107a94b926b4518baaf9bacd04d2f46" modify_external_catalogs: true
I20250110 20:57:07.291980 5595 raft_consensus.cc:2267] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:07.293934 5595 ts_tablet_manager.cc:1905] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:07.310568 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_bb34f3dffd1d417690819599206a7a92" } new_table_name: "default.Table_e470babc3c014ea5bba249532d50babf" modify_external_catalogs: true
I20250110 20:57:07.314613 5597 tablet_bootstrap.cc:492] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:07.314983 5597 ts_tablet_manager.cc:1397] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.005s
I20250110 20:57:07.318076 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet da24904dece04a6194790054f6d7153a with delete_type TABLET_DATA_DELETED (replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.317831 5597 raft_consensus.cc:357] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.318435 5597 raft_consensus.cc:383] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.318658 5597 raft_consensus.cc:738] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.319196 5597 consensus_queue.cc:260] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.321089 5597 ts_tablet_manager.cc:1428] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250110 20:57:07.321880 5597 tablet_bootstrap.cc:492] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:07.327416 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7a09ea54e79485b9892d52d0e04da85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.340788 5597 tablet_bootstrap.cc:654] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.346278 5594 tablet_bootstrap.cc:492] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:07.346660 5594 ts_tablet_manager.cc:1397] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.205s user 0.026s sys 0.017s
I20250110 20:57:07.348742 5594 raft_consensus.cc:357] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:07.349380 5594 raft_consensus.cc:383] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.349640 5594 raft_consensus.cc:738] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.350209 5594 consensus_queue.cc:260] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:07.353204 5746 raft_consensus.cc:491] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.353587 5746 raft_consensus.cc:513] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.355239 5746 leader_election.cc:290] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:07.356786 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 41de354807004e26bfb147f1d420037e (DEFAULT_TABLE table=default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.357038 5594 ts_tablet_manager.cc:1428] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20250110 20:57:07.358489 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e38b659f218d42298cb36752b5ca603d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:07.360749 5436 tablet_service.cc:1467] Processing CreateTablet for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (DEFAULT_TABLE table=default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.363993 5438 tablet_service.cc:1467] Processing CreateTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a (DEFAULT_TABLE table=default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.365371 5435 tablet_service.cc:1467] Processing CreateTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 (DEFAULT_TABLE table=default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.365998 5297 tablet_service.cc:1467] Processing CreateTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a (DEFAULT_TABLE table=default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:07.367092 5398 leader_election.cc:343] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: Tablet not RUNNING: STOPPED
W20250110 20:57:07.368077 5399 leader_election.cc:343] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: e38b659f218d42298cb36752b5ca603d
I20250110 20:57:07.368436 5399 leader_election.cc:304] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:07.369441 5597 tablet_bootstrap.cc:492] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:07.369797 5597 ts_tablet_manager.cc:1397] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.048s user 0.010s sys 0.001s
I20250110 20:57:07.371920 5597 raft_consensus.cc:357] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.372504 5597 raft_consensus.cc:383] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.372745 5597 raft_consensus.cc:738] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.373438 5597 consensus_queue.cc:260] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.374383 5600 tablet_bootstrap.cc:654] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.374888 5746 raft_consensus.cc:2743] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:07.376232 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41de354807004e26bfb147f1d420037e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.376339 5594 tablet_bootstrap.cc:492] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:07.380000 4898 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 2967c06715e241dcbbe784eed2373704
I20250110 20:57:07.381847 5594 tablet_bootstrap.cc:654] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.382889 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 2967c06715e241dcbbe784eed2373704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.383491 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet 2967c06715e241dcbbe784eed2373704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:07.375633 5597 ts_tablet_manager.cc:1428] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250110 20:57:07.387835 5597 tablet_bootstrap.cc:492] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:07.384478 5746 raft_consensus.cc:491] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.390089 5746 raft_consensus.cc:513] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.392513 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 (DEFAULT_TABLE table=default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.397868 5594 tablet_bootstrap.cc:492] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:07.398329 5594 ts_tablet_manager.cc:1397] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.000s
I20250110 20:57:07.400506 5594 raft_consensus.cc:357] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.401113 5594 raft_consensus.cc:383] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:07.401109 5398 leader_election.cc:343] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 70ac7d62f978433dad004f2908c4bbc4
I20250110 20:57:07.401502 5594 raft_consensus.cc:738] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.402019 5746 leader_election.cc:290] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:07.402035 5594 consensus_queue.cc:260] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.411002 5594 ts_tablet_manager.cc:1428] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20250110 20:57:07.412189 5594 tablet_bootstrap.cc:492] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:07.415544 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ac7d62f978433dad004f2908c4bbc4" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:07.416160 5176 raft_consensus.cc:2463] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:07.417104 5594 tablet_bootstrap.cc:654] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.421813 5399 leader_election.cc:304] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:07.424003 5769 raft_consensus.cc:2798] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:07.424300 5769 raft_consensus.cc:491] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:07.424569 5769 raft_consensus.cc:3054] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.428309 5597 tablet_bootstrap.cc:654] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.427230 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a809544a8ed648619ae05e19bdd40599"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:07.430619 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a809544a8ed648619ae05e19bdd40599 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:07.438155 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.439327 5600 tablet_bootstrap.cc:492] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.439813 5600 ts_tablet_manager.cc:1397] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.206s user 0.010s sys 0.000s
I20250110 20:57:07.442248 5600 raft_consensus.cc:357] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.448323 5600 raft_consensus.cc:383] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.448659 5600 raft_consensus.cc:738] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.452098 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.452659 5435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d67e94c8f9a414499b5e20a5cc0ead3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.452018 5595 ts_tablet_manager.cc:1918] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:07.457762 5595 log.cc:1198] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/e38b659f218d42298cb36752b5ca603d
I20250110 20:57:07.458395 5595 ts_tablet_manager.cc:1939] T e38b659f218d42298cb36752b5ca603d P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:07.459614 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d67e94c8f9a414499b5e20a5cc0ead3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.480427 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2cbb46ea11c4e7d8bc5993ccdc52247. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.493122 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet e38b659f218d42298cb36752b5ca603d (table default.Table_4a2a0aff7e744f57bc2c98b76bb0a199 [id=ba089f4f57284f80ac3621a88610fec1]) successfully deleted
W20250110 20:57:07.493940 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 2967c06715e241dcbbe784eed2373704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2967c06715e241dcbbe784eed2373704 already in progress: creating tablet
I20250110 20:57:07.496765 5600 consensus_queue.cc:260] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.499198 5600 ts_tablet_manager.cc:1428] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.059s user 0.003s sys 0.002s
I20250110 20:57:07.500056 5600 tablet_bootstrap.cc:492] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.502072 5739 raft_consensus.cc:491] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.502512 5739 raft_consensus.cc:513] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.504325 5739 leader_election.cc:290] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:07.506103 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472174c6885b446a839721157914f047" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:07.506769 5463 raft_consensus.cc:2463] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:07.508219 5110 leader_election.cc:304] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:07.510655 5739 raft_consensus.cc:2798] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:07.511004 5739 raft_consensus.cc:491] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:07.511273 5739 raft_consensus.cc:3054] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.514389 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472174c6885b446a839721157914f047" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:07.515043 5323 raft_consensus.cc:2463] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:07.527772 4899 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:07.534245 5600 tablet_bootstrap.cc:654] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.536667 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_638a8034373e4e3bbfb842beaacdaab8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:07.538671 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_638a8034373e4e3bbfb842beaacdaab8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:07.540655 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:07.542205 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet f0b5ba9f38c1493bba01180744f080e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0b5ba9f38c1493bba01180744f080e1 already in progress: creating tablet
W20250110 20:57:07.542771 4887 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 -> Peer 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233): Couldn't send request to peer 14d1e2d5c72f498f915c367e8750361f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250110 20:57:07.550058 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.556886 5769 raft_consensus.cc:513] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.557060 5739 raft_consensus.cc:513] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.558863 5769 leader_election.cc:290] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:07.560111 5398 leader_election.cc:343] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 70ac7d62f978433dad004f2908c4bbc4
I20250110 20:57:07.563678 5739 leader_election.cc:290] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:07.564695 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70ac7d62f978433dad004f2908c4bbc4" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:07.565217 5176 raft_consensus.cc:3054] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.565570 5600 tablet_bootstrap.cc:492] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.565935 5600 ts_tablet_manager.cc:1397] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.066s user 0.012s sys 0.000s
I20250110 20:57:07.567970 5600 raft_consensus.cc:357] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:07.568569 5600 raft_consensus.cc:383] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.568796 5600 raft_consensus.cc:738] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.569388 5600 consensus_queue.cc:260] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:07.572139 5600 ts_tablet_manager.cc:1428] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250110 20:57:07.572991 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472174c6885b446a839721157914f047" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:07.573397 5600 tablet_bootstrap.cc:492] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.578709 5739 raft_consensus.cc:491] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.579311 5463 raft_consensus.cc:3054] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.579113 5739 raft_consensus.cc:513] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.581183 5739 leader_election.cc:290] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:07.582449 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9155a626d26541ce9d0cf31508f19f15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:07.583011 5461 raft_consensus.cc:2463] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:07.584988 5110 leader_election.cc:304] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
W20250110 20:57:07.587718 5111 leader_election.cc:343] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 9155a626d26541ce9d0cf31508f19f15
I20250110 20:57:07.587883 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "472174c6885b446a839721157914f047" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:07.588943 5323 raft_consensus.cc:3054] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.590907 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e7ffb063594128ba137f9aea86bf15 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.592187 4895 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b115c55a632e444795c1dfc779b62d48" } modify_external_catalogs: true
I20250110 20:57:07.593587 4895 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b115c55a632e444795c1dfc779b62d48" } modify_external_catalogs: true
I20250110 20:57:07.595260 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet da24904dece04a6194790054f6d7153a with delete_type TABLET_DATA_DELETED (replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:07.598762 5600 tablet_bootstrap.cc:654] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.589238 5299 tablet_service.cc:1467] Processing CreateTablet for tablet 41de354807004e26bfb147f1d420037e (DEFAULT_TABLE table=default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.607522 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41de354807004e26bfb147f1d420037e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.615731 5600 tablet_bootstrap.cc:492] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.619256 5600 ts_tablet_manager.cc:1397] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.007s
I20250110 20:57:07.621349 5600 raft_consensus.cc:357] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.621996 5600 raft_consensus.cc:383] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.622248 5600 raft_consensus.cc:738] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.622782 5600 consensus_queue.cc:260] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.624744 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 41de354807004e26bfb147f1d420037e (DEFAULT_TABLE table=default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.625725 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41de354807004e26bfb147f1d420037e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.628157 5600 ts_tablet_manager.cc:1428] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250110 20:57:07.628947 5600 tablet_bootstrap.cc:492] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.608506 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 5793d909c6364047abc85ea5f993c521
I20250110 20:57:07.633482 5435 tablet_service.cc:1514] Processing DeleteTablet for tablet 5793d909c6364047abc85ea5f993c521 with delete_type TABLET_DATA_DELETED (replaced by tablet fcc7b1acf4114bf69d801437855760df) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:07.633926 4896 catalog_manager.cc:6596] ReplaceTablet: tablet 5793d909c6364047abc85ea5f993c521 deleted and replaced by tablet fcc7b1acf4114bf69d801437855760df
W20250110 20:57:07.613755 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f1fafed70adb496d97acfd5fea356292 consensus state has no leader
I20250110 20:57:07.637665 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet 5793d909c6364047abc85ea5f993c521 with delete_type TABLET_DATA_DELETED (replaced by tablet fcc7b1acf4114bf69d801437855760df) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:07.638674 5595 tablet_replica.cc:331] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:07.639302 5595 raft_consensus.cc:2238] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:07.639710 5595 raft_consensus.cc:2267] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:07.641736 5595 ts_tablet_manager.cc:1905] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:07.642391 5600 tablet_bootstrap.cc:654] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.661005 5600 tablet_bootstrap.cc:492] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.661373 5600 ts_tablet_manager.cc:1397] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.033s user 0.007s sys 0.005s
I20250110 20:57:07.661710 5463 raft_consensus.cc:2463] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:07.662039 5153 tablet_service.cc:1467] Processing CreateTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a (DEFAULT_TABLE table=default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.662626 5323 raft_consensus.cc:2463] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:07.663489 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.663472 5600 raft_consensus.cc:357] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.664119 5600 raft_consensus.cc:383] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.664193 5111 leader_election.cc:304] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:07.664366 5600 raft_consensus.cc:738] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.665099 5600 consensus_queue.cc:260] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.670810 5600 ts_tablet_manager.cc:1428] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250110 20:57:07.671571 5600 tablet_bootstrap.cc:492] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.671581 5176 raft_consensus.cc:2463] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:07.681946 5777 raft_consensus.cc:2798] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:07.682312 5777 raft_consensus.cc:491] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:07.686465 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet 2967c06715e241dcbbe784eed2373704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:07.686493 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 (DEFAULT_TABLE table=default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.688838 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d67e94c8f9a414499b5e20a5cc0ead3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.689664 5600 tablet_bootstrap.cc:654] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.703173 5780 raft_consensus.cc:2798] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:07.703637 5780 raft_consensus.cc:695] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:07.704694 5777 raft_consensus.cc:3054] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.717705 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:07.718616 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet 5793d909c6364047abc85ea5f993c521 with delete_type TABLET_DATA_DELETED (replaced by tablet fcc7b1acf4114bf69d801437855760df) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:07.720291 5773 tablet_replica.cc:331] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:07.721061 5773 raft_consensus.cc:2238] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:07.721566 5773 raft_consensus.cc:2267] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:07.704370 5780 consensus_queue.cc:237] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.736220 4900 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5107a94b926b4518baaf9bacd04d2f46" } modify_external_catalogs: true
I20250110 20:57:07.736776 4900 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5107a94b926b4518baaf9bacd04d2f46" } modify_external_catalogs: true
W20250110 20:57:07.741343 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet da24904dece04a6194790054f6d7153a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da24904dece04a6194790054f6d7153a already in progress: creating tablet
I20250110 20:57:07.741911 5773 ts_tablet_manager.cc:1905] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:07.742586 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 2967c06715e241dcbbe784eed2373704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2967c06715e241dcbbe784eed2373704 already in progress: creating tablet
I20250110 20:57:07.748068 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ac07ac6fc9f477ba7f9accc5533be39 from {username='slave'} at 127.0.0.1:36568
W20250110 20:57:07.753193 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 89078.5us
I20250110 20:57:07.754179 5399 leader_election.cc:304] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:07.754935 5600 tablet_bootstrap.cc:492] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.755216 5769 raft_consensus.cc:2798] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:07.755311 5600 ts_tablet_manager.cc:1397] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.084s user 0.008s sys 0.013s
I20250110 20:57:07.755602 5769 raft_consensus.cc:695] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:07.756311 5769 consensus_queue.cc:237] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.757349 5600 raft_consensus.cc:357] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.758257 5600 raft_consensus.cc:383] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.758497 5600 raft_consensus.cc:738] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.759140 5600 consensus_queue.cc:260] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.760937 5595 ts_tablet_manager.cc:1918] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:07.760937 5592 ts_tablet_manager.cc:1918] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:07.761270 5595 log.cc:1198] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/5793d909c6364047abc85ea5f993c521
I20250110 20:57:07.761270 5592 log.cc:1198] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/29599cfce9ba4875b7aed08f5094ce21
I20250110 20:57:07.761888 5595 ts_tablet_manager.cc:1939] T 5793d909c6364047abc85ea5f993c521 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:07.761886 5592 ts_tablet_manager.cc:1939] T 29599cfce9ba4875b7aed08f5094ce21 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:07.764194 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 29599cfce9ba4875b7aed08f5094ce21 (table default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f [id=3bbd1acd709b44899907148df944b20d]) successfully deleted
I20250110 20:57:07.765022 5592 tablet_replica.cc:331] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:07.766731 5592 raft_consensus.cc:2238] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:07.767135 5592 raft_consensus.cc:2267] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:07.768445 5463 raft_consensus.cc:3054] T 5256d081843a492d8081c4319f3446d5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.768554 5600 ts_tablet_manager.cc:1428] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250110 20:57:07.769266 5592 ts_tablet_manager.cc:1905] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:07.769371 5600 tablet_bootstrap.cc:492] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.777181 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 5793d909c6364047abc85ea5f993c521 (table default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]) successfully deleted
I20250110 20:57:07.782934 5600 tablet_bootstrap.cc:654] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:07.787001 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet f0b5ba9f38c1493bba01180744f080e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0b5ba9f38c1493bba01180744f080e1 already in progress: creating tablet
W20250110 20:57:07.796363 4886 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d310cf52aa8c4930a7821117a46dc958 consensus state has no leader
I20250110 20:57:07.799088 5600 tablet_bootstrap.cc:492] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.799463 5600 ts_tablet_manager.cc:1397] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20250110 20:57:07.801685 5600 raft_consensus.cc:357] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.802546 5600 raft_consensus.cc:383] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.802795 5600 raft_consensus.cc:738] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.803386 5600 consensus_queue.cc:260] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.808847 5600 ts_tablet_manager.cc:1428] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250110 20:57:07.809600 5600 tablet_bootstrap.cc:492] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.817633 5600 tablet_bootstrap.cc:654] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.820341 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d40a02bf99094290afe8462ab6a3672f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:07.821755 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d40a02bf99094290afe8462ab6a3672f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:07.829478 5779 raft_consensus.cc:491] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.829981 5779 raft_consensus.cc:513] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.831933 5779 leader_election.cc:290] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:07.833540 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e7ffb063594128ba137f9aea86bf15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:07.834131 5461 raft_consensus.cc:2463] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:07.834468 5597 tablet_bootstrap.cc:492] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:07.834812 5597 ts_tablet_manager.cc:1397] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.447s user 0.023s sys 0.042s
I20250110 20:57:07.837206 5594 tablet_bootstrap.cc:492] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:07.837635 5594 ts_tablet_manager.cc:1397] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.426s user 0.040s sys 0.020s
I20250110 20:57:07.839380 5600 tablet_bootstrap.cc:492] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.839730 5600 ts_tablet_manager.cc:1397] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.030s user 0.009s sys 0.002s
I20250110 20:57:07.839980 5594 raft_consensus.cc:357] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.841693 5600 raft_consensus.cc:357] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.843184 5597 raft_consensus.cc:357] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.843777 5597 raft_consensus.cc:383] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.844007 5597 raft_consensus.cc:738] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.844514 5597 consensus_queue.cc:260] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:07.845710 5110 leader_election.cc:304] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
W20250110 20:57:07.846565 5111 leader_election.cc:343] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: a7e7ffb063594128ba137f9aea86bf15
I20250110 20:57:07.847364 5779 raft_consensus.cc:2798] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:07.847721 5779 raft_consensus.cc:491] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:07.848081 5779 raft_consensus.cc:3054] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.842343 5600 raft_consensus.cc:383] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.849359 5600 raft_consensus.cc:738] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.850435 5600 consensus_queue.cc:260] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:07.855285 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7a09ea54e79485b9892d52d0e04da85 consensus state has no leader
I20250110 20:57:07.862640 5600 ts_tablet_manager.cc:1428] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250110 20:57:07.865046 5600 tablet_bootstrap.cc:492] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.872676 5739 consensus_queue.cc:1035] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:07.883066 5600 tablet_bootstrap.cc:654] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.888769 5713 raft_consensus.cc:491] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:07.889256 5713 raft_consensus.cc:513] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:07.891256 5713 leader_election.cc:290] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:07.901122 5600 tablet_bootstrap.cc:492] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:07.901752 5600 ts_tablet_manager.cc:1397] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.000s
I20250110 20:57:07.902555 5597 ts_tablet_manager.cc:1428] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
I20250110 20:57:07.902380 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78538b2e7bcc45c7afaa145bad6e5813" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:07.903370 5597 tablet_bootstrap.cc:492] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:07.905577 5600 raft_consensus.cc:357] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.906205 5600 raft_consensus.cc:383] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:07.906453 5600 raft_consensus.cc:738] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:07.906994 5600 consensus_queue.cc:260] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.927606 4891 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d0b2c46a454745699731c3406fc3ce90" } new_table_name: "default.Table_447ad91cdb774363a84b85d71a154011" modify_external_catalogs: true
W20250110 20:57:07.928934 5257 leader_election.cc:343] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:07.937796 5600 ts_tablet_manager.cc:1428] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.036s user 0.005s sys 0.002s
I20250110 20:57:07.940587 5600 tablet_bootstrap.cc:492] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:07.946107 5739 consensus_queue.cc:1035] T 5256d081843a492d8081c4319f3446d5 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:07.948495 5597 tablet_bootstrap.cc:654] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.963434 5600 tablet_bootstrap.cc:654] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:07.969952 5773 ts_tablet_manager.cc:1918] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:07.970332 5773 log.cc:1198] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/5793d909c6364047abc85ea5f993c521
I20250110 20:57:07.970996 5773 ts_tablet_manager.cc:1939] T 5793d909c6364047abc85ea5f993c521 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:07.973134 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 8fe93f1a08f3432dba9c6ce76b4c99b0
I20250110 20:57:07.975051 5779 raft_consensus.cc:513] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:07.976873 5779 leader_election.cc:290] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:07.978763 5299 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:07.979755 5111 leader_election.cc:343] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: a7e7ffb063594128ba137f9aea86bf15
I20250110 20:57:07.981093 4899 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:07.981240 5595 tablet_replica.cc:331] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:07.892236 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78538b2e7bcc45c7afaa145bad6e5813" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:07.969226 5592 ts_tablet_manager.cc:1918] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:07.983991 5592 log.cc:1198] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/b26f58b066f646e5b11ddf20cbbf9e1a
I20250110 20:57:07.983841 5149 tablet_service.cc:1467] Processing CreateTablet for tablet d1afe41a09954fc8873ea53a330ea03f (DEFAULT_TABLE table=default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c [id=0f2abb2b06c2485aab55a063ad4a0b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.984606 5592 ts_tablet_manager.cc:1939] T b26f58b066f646e5b11ddf20cbbf9e1a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:07.984830 5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1afe41a09954fc8873ea53a330ea03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.986749 5592 tablet_replica.cc:331] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:07.979892 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7e7ffb063594128ba137f9aea86bf15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:07.987424 5464 raft_consensus.cc:3054] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:07.980158 5436 tablet_service.cc:1467] Processing CreateTablet for tablet d1afe41a09954fc8873ea53a330ea03f (DEFAULT_TABLE table=default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c [id=0f2abb2b06c2485aab55a063ad4a0b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.990178 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1afe41a09954fc8873ea53a330ea03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.990268 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 5793d909c6364047abc85ea5f993c521 (table default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]) successfully deleted
I20250110 20:57:07.991125 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet b26f58b066f646e5b11ddf20cbbf9e1a (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]) successfully deleted
W20250110 20:57:07.983515 5255 leader_election.cc:343] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:07.993562 5255 leader_election.cc:304] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:07.994832 5713 raft_consensus.cc:2743] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:07.995544 5147 tablet_service.cc:1467] Processing CreateTablet for tablet cd0a666892d04e97a4b9967e38ab0946 (DEFAULT_TABLE table=default.Table_c3b44e7a68ac47a1ad0f7c5496130418 [id=143a11fac7564c92b6a7224e6ac31bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.996168 5437 tablet_service.cc:1467] Processing CreateTablet for tablet c951d5ca47e8409db646b43bb04a7034 (DEFAULT_TABLE table=default.Table_f8573af69b094101bbd0f64ce7c1716c [id=723ca40b46694de9802850cb89dca66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:07.996542 5147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0a666892d04e97a4b9967e38ab0946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:07.997151 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c951d5ca47e8409db646b43bb04a7034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.006243 5435 tablet_service.cc:1467] Processing CreateTablet for tablet cd0a666892d04e97a4b9967e38ab0946 (DEFAULT_TABLE table=default.Table_c3b44e7a68ac47a1ad0f7c5496130418 [id=143a11fac7564c92b6a7224e6ac31bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.006572 5592 raft_consensus.cc:2238] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:08.007237 5435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0a666892d04e97a4b9967e38ab0946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.009954 5600 tablet_bootstrap.cc:492] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.009865 5151 tablet_service.cc:1467] Processing CreateTablet for tablet c6914e258314423c859eb31e85853274 (DEFAULT_TABLE table=default.Table_a1f70418cc2f433d90434c4621fd49d0 [id=a02df045e79245db9b8b7ad158771621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.010336 5600 ts_tablet_manager.cc:1397] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.001s
I20250110 20:57:08.010914 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6914e258314423c859eb31e85853274. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.011389 5150 tablet_service.cc:1467] Processing CreateTablet for tablet c951d5ca47e8409db646b43bb04a7034 (DEFAULT_TABLE table=default.Table_f8573af69b094101bbd0f64ce7c1716c [id=723ca40b46694de9802850cb89dca66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.012204 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c951d5ca47e8409db646b43bb04a7034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.014124 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 527bbddda8184a6eb67c2411ae50db63 (DEFAULT_TABLE table=default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.014155 5441 tablet_service.cc:1467] Processing CreateTablet for tablet c6914e258314423c859eb31e85853274 (DEFAULT_TABLE table=default.Table_a1f70418cc2f433d90434c4621fd49d0 [id=a02df045e79245db9b8b7ad158771621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.014860 5595 raft_consensus.cc:2238] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:08.015157 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527bbddda8184a6eb67c2411ae50db63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.015370 5595 raft_consensus.cc:2267] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.015427 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 527bbddda8184a6eb67c2411ae50db63 (DEFAULT_TABLE table=default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.016328 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527bbddda8184a6eb67c2411ae50db63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.017215 5595 ts_tablet_manager.cc:1905] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:08.017148 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1ca1670916401ab0a43685df1b6759 (DEFAULT_TABLE table=default.Table_792d0af8e9d04ad28d950c7656211372 [id=176f78029038442b932e8c92372948ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.018841 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1ca1670916401ab0a43685df1b6759 (DEFAULT_TABLE table=default.Table_792d0af8e9d04ad28d950c7656211372 [id=176f78029038442b932e8c92372948ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.019798 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1ca1670916401ab0a43685df1b6759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.022387 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1ca1670916401ab0a43685df1b6759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.022816 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:08.025939 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6914e258314423c859eb31e85853274. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.028949 5442 tablet_service.cc:1467] Processing CreateTablet for tablet fde470279464496aa7efdee3800754f3 (DEFAULT_TABLE table=default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.029846 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fde470279464496aa7efdee3800754f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.030261 5152 tablet_service.cc:1467] Processing CreateTablet for tablet fde470279464496aa7efdee3800754f3 (DEFAULT_TABLE table=default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.031302 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fde470279464496aa7efdee3800754f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.007225 5592 raft_consensus.cc:2267] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.035405 5592 ts_tablet_manager.cc:1905] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:08.039712 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 491626848ad54be3a18186c65d6dbb60 (DEFAULT_TABLE table=default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.040720 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 491626848ad54be3a18186c65d6dbb60. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:08.043105 5111 consensus_peers.cc:487] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:08.012765 5600 raft_consensus.cc:357] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.049919 5600 raft_consensus.cc:383] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.050207 5600 raft_consensus.cc:738] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.058524 5600 consensus_queue.cc:260] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.052490 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 477b09061a4a4f278720168872a83258 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:08.065505 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_b4ab0cd248e44e67ab389fe94e4a1bab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:08.067935 5464 raft_consensus.cc:2463] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:08.055701 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_46cf92641ac847ac88303d12596df78d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:08.073966 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4ab0cd248e44e67ab389fe94e4a1bab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:08.074018 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_61b3b8a681344e25acecd34b7455bb08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:08.075392 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b3b8a681344e25acecd34b7455bb08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:08.076140 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46cf92641ac847ac88303d12596df78d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:08.082463 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1d485a05ae354e4492c99bf9c67d6913" } new_table_name: "default.Table_b15b46fb1ed048c8b2057b0ab88797cd" modify_external_catalogs: true
I20250110 20:57:08.085517 5438 tablet_service.cc:1467] Processing CreateTablet for tablet 491626848ad54be3a18186c65d6dbb60 (DEFAULT_TABLE table=default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.086632 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 491626848ad54be3a18186c65d6dbb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.087272 5437 tablet_service.cc:1467] Processing CreateTablet for tablet bd9d0249798e4f24adf0b36d9e059b7e (DEFAULT_TABLE table=default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.090632 5600 ts_tablet_manager.cc:1428] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.080s user 0.005s sys 0.001s
I20250110 20:57:08.088824 5444 tablet_service.cc:1467] Processing CreateTablet for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (DEFAULT_TABLE table=default.Table_1ec05f4286b94b13bbe215ff5cba4d32 [id=2c78ee52e44a46fe877f474a7ecb771b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.090544 5443 tablet_service.cc:1467] Processing CreateTablet for tablet da6a9a054aa54455b4979e4c633ad6b4 (DEFAULT_TABLE table=default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.092291 5434 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb (DEFAULT_TABLE table=default.Table_b89da7c5e86341c39853bbacce851ad4 [id=7833678313fa4c76ad1eb6d19315bb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.093974 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9d0249798e4f24adf0b36d9e059b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.095805 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da6a9a054aa54455b4979e4c633ad6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.096351 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7f09df8e274f3e9542292d5c1ec6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.096895 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.094292 5594 raft_consensus.cc:383] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.098843 5594 raft_consensus.cc:738] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.099622 5299 tablet_service.cc:1467] Processing CreateTablet for tablet d1afe41a09954fc8873ea53a330ea03f (DEFAULT_TABLE table=default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c [id=0f2abb2b06c2485aab55a063ad4a0b47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.100776 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1afe41a09954fc8873ea53a330ea03f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.091565 5600 tablet_bootstrap.cc:492] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.105244 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:08.112031 5295 tablet_service.cc:1467] Processing CreateTablet for tablet c951d5ca47e8409db646b43bb04a7034 (DEFAULT_TABLE table=default.Table_f8573af69b094101bbd0f64ce7c1716c [id=723ca40b46694de9802850cb89dca66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.113106 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c951d5ca47e8409db646b43bb04a7034. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.107777 5594 consensus_queue.cc:260] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:08.110597 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 52279.3us
I20250110 20:57:08.121122 5773 tablet_replica.cc:331] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:08.121910 5773 raft_consensus.cc:2238] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:08.122660 5146 tablet_service.cc:1467] Processing CreateTablet for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (DEFAULT_TABLE table=default.Table_1ec05f4286b94b13bbe215ff5cba4d32 [id=2c78ee52e44a46fe877f474a7ecb771b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.124339 5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7f09df8e274f3e9542292d5c1ec6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.126966 5297 tablet_service.cc:1467] Processing CreateTablet for tablet cd0a666892d04e97a4b9967e38ab0946 (DEFAULT_TABLE table=default.Table_c3b44e7a68ac47a1ad0f7c5496130418 [id=143a11fac7564c92b6a7224e6ac31bfb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.128005 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd0a666892d04e97a4b9967e38ab0946. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.128880 5600 tablet_bootstrap.cc:654] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.134037 5156 tablet_service.cc:1467] Processing CreateTablet for tablet bd9d0249798e4f24adf0b36d9e059b7e (DEFAULT_TABLE table=default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.134997 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9d0249798e4f24adf0b36d9e059b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.133152 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb (DEFAULT_TABLE table=default.Table_b89da7c5e86341c39853bbacce851ad4 [id=7833678313fa4c76ad1eb6d19315bb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.142963 5300 tablet_service.cc:1467] Processing CreateTablet for tablet c6914e258314423c859eb31e85853274 (DEFAULT_TABLE table=default.Table_a1f70418cc2f433d90434c4621fd49d0 [id=a02df045e79245db9b8b7ad158771621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.143980 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6914e258314423c859eb31e85853274. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.162817 5600 tablet_bootstrap.cc:492] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.163247 5600 ts_tablet_manager.cc:1397] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.001s
I20250110 20:57:08.163980 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 527bbddda8184a6eb67c2411ae50db63 (DEFAULT_TABLE table=default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.164997 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 527bbddda8184a6eb67c2411ae50db63. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.165661 5594 ts_tablet_manager.cc:1428] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.328s user 0.004s sys 0.001s
I20250110 20:57:08.168310 5600 raft_consensus.cc:357] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.168943 5600 raft_consensus.cc:383] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.169181 5600 raft_consensus.cc:738] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.119082 5777 raft_consensus.cc:513] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.169847 5600 consensus_queue.cc:260] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.171911 5600 ts_tablet_manager.cc:1428] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20250110 20:57:08.171929 5773 pending_rounds.cc:62] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:08.172168 5773 pending_rounds.cc:69] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878604151046144 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "8fe93f1a08f3432dba9c6ce76b4c99b0" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_b115c55a632e444795c1dfc779b62d48" dest_uuid: "ac9003be04f340f3886b5625cc928eb0" new_extra_config { } }
I20250110 20:57:08.172698 5600 tablet_bootstrap.cc:492] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:08.173137 5773 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35166: Aborted: Op aborted
I20250110 20:57:08.181562 4893 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_653008e13c0c4667a68753a92ae1a596" } modify_external_catalogs: true
I20250110 20:57:08.182132 4893 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_653008e13c0c4667a68753a92ae1a596" } modify_external_catalogs: true
I20250110 20:57:08.183177 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_5f0518ca447a401eb8fe8655ecdc9a55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:08.184725 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f0518ca447a401eb8fe8655ecdc9a55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:08.127156 5110 leader_election.cc:304] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
W20250110 20:57:08.191912 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 81204.6us
I20250110 20:57:08.193264 5779 raft_consensus.cc:2798] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:08.193809 5779 raft_consensus.cc:695] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:08.194655 5779 consensus_queue.cc:237] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.195256 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 2c1ca1670916401ab0a43685df1b6759 (DEFAULT_TABLE table=default.Table_792d0af8e9d04ad28d950c7656211372 [id=176f78029038442b932e8c92372948ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.196296 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c1ca1670916401ab0a43685df1b6759. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.197952 5600 tablet_bootstrap.cc:654] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.208096 5145 tablet_service.cc:1467] Processing CreateTablet for tablet da6a9a054aa54455b4979e4c633ad6b4 (DEFAULT_TABLE table=default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.211414 5298 tablet_service.cc:1467] Processing CreateTablet for tablet fde470279464496aa7efdee3800754f3 (DEFAULT_TABLE table=default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.212441 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fde470279464496aa7efdee3800754f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.223098 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 491626848ad54be3a18186c65d6dbb60 (DEFAULT_TABLE table=default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.224248 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 491626848ad54be3a18186c65d6dbb60. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.227154 5302 tablet_service.cc:1467] Processing CreateTablet for tablet bd9d0249798e4f24adf0b36d9e059b7e (DEFAULT_TABLE table=default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.228252 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9d0249798e4f24adf0b36d9e059b7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.242558 5777 leader_election.cc:290] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:08.249028 5773 raft_consensus.cc:2267] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.249971 5294 tablet_service.cc:1467] Processing CreateTablet for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (DEFAULT_TABLE table=default.Table_1ec05f4286b94b13bbe215ff5cba4d32 [id=2c78ee52e44a46fe877f474a7ecb771b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.253034 5773 ts_tablet_manager.cc:1905] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:08.263864 5111 leader_election.cc:343] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 9155a626d26541ce9d0cf31508f19f15
I20250110 20:57:08.264179 5600 tablet_bootstrap.cc:492] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.264536 5600 ts_tablet_manager.cc:1397] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.000s
I20250110 20:57:08.273830 5293 tablet_service.cc:1467] Processing CreateTablet for tablet da6a9a054aa54455b4979e4c633ad6b4 (DEFAULT_TABLE table=default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.275252 5592 ts_tablet_manager.cc:1918] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:08.275620 5592 log.cc:1198] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/9155a626d26541ce9d0cf31508f19f15
I20250110 20:57:08.275377 5600 raft_consensus.cc:357] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:08.276033 5600 raft_consensus.cc:383] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.276217 5592 ts_tablet_manager.cc:1939] T 9155a626d26541ce9d0cf31508f19f15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:08.276283 5600 raft_consensus.cc:738] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.276861 5600 consensus_queue.cc:260] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:08.282333 4892 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:08.283067 4892 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 78538b2e7bcc45c7afaa145bad6e5813 (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]). Expected version 2 got 0
I20250110 20:57:08.286005 5600 ts_tablet_manager.cc:1428] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250110 20:57:08.292066 5805 raft_consensus.cc:491] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.292503 5805 raft_consensus.cc:513] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.294284 5805 leader_election.cc:290] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.295467 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb7f09df8e274f3e9542292d5c1ec6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.321921 5600 tablet_bootstrap.cc:492] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.287216 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb (DEFAULT_TABLE table=default.Table_b89da7c5e86341c39853bbacce851ad4 [id=7833678313fa4c76ad1eb6d19315bb23]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.323168 5713 raft_consensus.cc:491] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.323994 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.327903 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da6a9a054aa54455b4979e4c633ad6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.327653 5713 raft_consensus.cc:513] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:08.333060 5257 leader_election.cc:343] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.307636 5805 raft_consensus.cc:491] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.333705 5805 raft_consensus.cc:513] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.334132 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.342254 5713 leader_election.cc:290] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.342676 5713 raft_consensus.cc:491] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.344192 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1fafed70adb496d97acfd5fea356292" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:08.345324 5257 leader_election.cc:343] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.350657 5597 tablet_bootstrap.cc:492] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:08.355150 5600 tablet_bootstrap.cc:654] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:08.355080 4892 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 78538b2e7bcc45c7afaa145bad6e5813 (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 78538b2e7bcc45c7afaa145bad6e5813 consensus state has no leader
I20250110 20:57:08.365185 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9155a626d26541ce9d0cf31508f19f15" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:08.367344 5805 leader_election.cc:290] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.368216 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1fafed70adb496d97acfd5fea356292" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.371132 5769 raft_consensus.cc:491] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.371520 5769 raft_consensus.cc:513] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.346696 5713 raft_consensus.cc:513] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:08.373435 5713 leader_election.cc:290] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:08.374397 5769 leader_election.cc:290] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.374997 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:08.377372 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a520230d9fd4a74914e46f784dcf747" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.378762 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.379240 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac07ac6fc9f477ba7f9accc5533be39" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.382131 5595 ts_tablet_manager.cc:1918] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 20:57:08.389292 5255 leader_election.cc:343] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.389694 5255 leader_election.cc:304] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:08.390631 5255 leader_election.cc:343] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:08.391184 5255 leader_election.cc:343] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:08.391685 5255 leader_election.cc:343] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.385270 5595 log.cc:1198] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/8fe93f1a08f3432dba9c6ce76b4c99b0
I20250110 20:57:08.391997 5255 leader_election.cc:304] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:08.392875 5595 ts_tablet_manager.cc:1939] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:08.392793 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b65488c884452d876ced1c3d1efb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:08.396256 5805 raft_consensus.cc:2743] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.396798 5713 raft_consensus.cc:2743] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.401516 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:08.406754 5398 leader_election.cc:343] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 477b09061a4a4f278720168872a83258
I20250110 20:57:08.408834 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 961fef96e9124aafb1c51e287f603b64 with delete_type TABLET_DATA_DELETED (replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:08.418709 5805 raft_consensus.cc:491] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.419137 5805 raft_consensus.cc:513] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.420864 5805 leader_election.cc:290] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:08.369120 5110 leader_election.cc:343] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:08.424896 5110 leader_election.cc:304] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
W20250110 20:57:08.426302 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 57213.4us
I20250110 20:57:08.382143 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477b09061a4a4f278720168872a83258" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:08.430684 5399 leader_election.cc:343] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.431171 5399 leader_election.cc:304] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:08.432854 5769 raft_consensus.cc:2743] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.433564 5777 raft_consensus.cc:2743] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.401676 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da6a9a054aa54455b4979e4c633ad6b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.437718 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.439172 5600 tablet_bootstrap.cc:492] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.439524 5600 ts_tablet_manager.cc:1397] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.143s user 0.010s sys 0.010s
I20250110 20:57:08.442639 5600 raft_consensus.cc:357] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.443318 5600 raft_consensus.cc:383] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.443569 5600 raft_consensus.cc:738] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.444188 5600 consensus_queue.cc:260] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.449774 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9776df98cb54be38a521852634490e7" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:08.450981 5255 leader_election.cc:343] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.453363 5595 tablet_replica.cc:331] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:08.454075 5595 raft_consensus.cc:2238] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:08.454501 5595 raft_consensus.cc:2267] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.455984 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 85) took 1334 ms. Trace:
I20250110 20:57:08.456290 5592 rpcz_store.cc:276] 0110 20:57:07.122062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.122195 (+ 133us) service_pool.cc:225] Handling call
0110 20:57:07.986620 (+864425us) ts_tablet_manager.cc:1145] Deleting tablet 9155a626d26541ce9d0cf31508f19f15
0110 20:57:07.986655 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:08.175768 (+189113us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:08.275048 (+ 99280us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:08.455928 (+180880us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":197095,"tablet-delete.queue_time_us":863913}
I20250110 20:57:08.456823 5595 ts_tablet_manager.cc:1905] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:08.456202 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]) successfully deleted
I20250110 20:57:08.458392 5592 tablet_replica.cc:331] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:08.459082 5592 raft_consensus.cc:2238] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:08.459651 5592 pending_rounds.cc:62] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:08.459801 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 9155a626d26541ce9d0cf31508f19f15 (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]) successfully deleted
I20250110 20:57:08.459861 5592 pending_rounds.cc:69] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878603298013184 op_type: NO_OP noop_request { }
I20250110 20:57:08.460170 5592 raft_consensus.cc:2883] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:08.460449 5592 raft_consensus.cc:2267] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.463012 5592 ts_tablet_manager.cc:1905] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:08.446892 5600 ts_tablet_manager.cc:1428] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20250110 20:57:08.465328 5600 tablet_bootstrap.cc:492] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.470057 5597 ts_tablet_manager.cc:1397] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.567s user 0.006s sys 0.005s
I20250110 20:57:08.472285 5597 raft_consensus.cc:357] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.473663 5594 tablet_bootstrap.cc:492] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:08.481091 5805 raft_consensus.cc:491] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.481534 5805 raft_consensus.cc:513] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.482616 5600 tablet_bootstrap.cc:654] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.483319 5805 leader_election.cc:290] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.485145 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b65488c884452d876ced1c3d1efb9a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:08.486511 5255 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.513872 5805 raft_consensus.cc:491] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.514317 5805 raft_consensus.cc:513] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.516665 5805 leader_election.cc:290] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.519059 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961fef96e9124aafb1c51e287f603b64" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:08.520157 5255 leader_election.cc:343] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.526058 5805 raft_consensus.cc:491] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 20:57:08.525843 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.252:37233, user_credentials={real_user=slave}} blocked reactor thread for 43827.7us
I20250110 20:57:08.526499 5805 raft_consensus.cc:513] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.529808 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d634add82994f74960cd2b8373a4dc6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:08.530956 5255 leader_election.cc:343] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.532332 5805 leader_election.cc:290] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.533088 5597 raft_consensus.cc:383] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.533476 5597 raft_consensus.cc:738] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.534276 5597 consensus_queue.cc:260] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:08.404989 5153 consensus_meta.cc:287] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 0.654s user 0.006s sys 0.000s
I20250110 20:57:08.535794 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a520230d9fd4a74914e46f784dcf747" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.537976 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac07ac6fc9f477ba7f9accc5533be39" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.538915 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9776df98cb54be38a521852634490e7" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.539855 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b65488c884452d876ced1c3d1efb9a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.540840 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961fef96e9124aafb1c51e287f603b64" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.542080 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d634add82994f74960cd2b8373a4dc6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:08.544199 5257 leader_election.cc:343] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.544564 5257 leader_election.cc:304] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:08.545782 5257 leader_election.cc:343] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.546125 5257 leader_election.cc:304] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:08.546888 5257 leader_election.cc:343] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.547211 5809 raft_consensus.cc:2743] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.547816 5809 raft_consensus.cc:2743] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.547201 5257 leader_election.cc:304] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:08.548597 5257 raft_consensus.cc:2681] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250110 20:57:08.549394 5257 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.549700 5257 leader_election.cc:304] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:08.550274 5713 raft_consensus.cc:2743] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:08.551160 5257 leader_election.cc:343] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.551534 5257 leader_election.cc:304] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:08.552147 5713 raft_consensus.cc:2743] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:08.553067 5257 leader_election.cc:343] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.553459 5257 leader_election.cc:304] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:08.554040 5713 raft_consensus.cc:2743] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.567019 5594 tablet_bootstrap.cc:654] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.584672 5597 ts_tablet_manager.cc:1428] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.107s user 0.005s sys 0.001s
W20250110 20:57:08.585768 4891 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7462d21b91c2441fb9e11383466e9ca7 consensus state has no leader
I20250110 20:57:08.591356 5600 tablet_bootstrap.cc:492] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.591747 5600 ts_tablet_manager.cc:1397] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.017s
I20250110 20:57:08.594280 5600 raft_consensus.cc:357] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.594928 5600 raft_consensus.cc:383] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.595180 5600 raft_consensus.cc:738] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.594854 5597 tablet_bootstrap.cc:492] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:08.595710 5600 consensus_queue.cc:260] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.607285 5600 ts_tablet_manager.cc:1428] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20250110 20:57:08.608104 5600 tablet_bootstrap.cc:492] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.608721 5594 tablet_bootstrap.cc:492] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:08.610337 5594 ts_tablet_manager.cc:1397] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.000s
I20250110 20:57:08.614012 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:08.615243 4896 catalog_manager.cc:6596] ReplaceTablet: tablet 1ac07ac6fc9f477ba7f9accc5533be39 deleted and replaced by tablet 371fb7408f0149c8b00659e09ebaf48a
I20250110 20:57:08.615419 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 with delete_type TABLET_DATA_DELETED (replaced by tablet 371fb7408f0149c8b00659e09ebaf48a) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:08.618291 5594 raft_consensus.cc:357] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.618930 5594 raft_consensus.cc:383] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.620507 5600 tablet_bootstrap.cc:654] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.622875 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 with delete_type TABLET_DATA_DELETED (replaced by tablet 371fb7408f0149c8b00659e09ebaf48a) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:08.623319 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 with delete_type TABLET_DATA_DELETED (replaced by tablet 371fb7408f0149c8b00659e09ebaf48a) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:08.625013 5597 tablet_bootstrap.cc:654] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.619158 5594 raft_consensus.cc:738] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.636691 5594 consensus_queue.cc:260] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.638103 5600 tablet_bootstrap.cc:492] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.638482 5600 ts_tablet_manager.cc:1397] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.004s
I20250110 20:57:08.647450 5713 raft_consensus.cc:491] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.647992 4897 master_service.cc:946] ReplaceTablet: received request to replace tablet 78538b2e7bcc45c7afaa145bad6e5813 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:08.647869 5713 raft_consensus.cc:513] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.643904 5600 raft_consensus.cc:357] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.648686 5600 raft_consensus.cc:383] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.648929 5600 raft_consensus.cc:738] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:08.648976 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e27e43461251413b97721323cd935374 consensus state has no leader
I20250110 20:57:08.649672 5713 leader_election.cc:290] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.649493 5600 consensus_queue.cc:260] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.651486 5600 ts_tablet_manager.cc:1428] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250110 20:57:08.651871 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e361fb8b47d74548b5a2326d25a9e28c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.657004 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e361fb8b47d74548b5a2326d25a9e28c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:08.657920 5257 leader_election.cc:343] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:08.658397 5255 leader_election.cc:343] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.658840 5600 tablet_bootstrap.cc:492] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.658766 5255 leader_election.cc:304] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:08.659952 5809 raft_consensus.cc:2743] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.660830 5595 ts_tablet_manager.cc:1918] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:08.661157 5595 log.cc:1198] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/d9776df98cb54be38a521852634490e7
I20250110 20:57:08.661747 5595 ts_tablet_manager.cc:1939] T d9776df98cb54be38a521852634490e7 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:08.662560 5592 ts_tablet_manager.cc:1918] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:08.662940 5592 log.cc:1198] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/70ac7d62f978433dad004f2908c4bbc4
I20250110 20:57:08.663600 5592 ts_tablet_manager.cc:1939] T 70ac7d62f978433dad004f2908c4bbc4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:08.665903 5595 tablet_replica.cc:331] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:08.665863 5779 raft_consensus.cc:491] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.666584 5595 raft_consensus.cc:2238] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:08.666268 5779 raft_consensus.cc:513] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.666988 5595 raft_consensus.cc:2267] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.667976 5779 leader_election.cc:290] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:08.668470 5595 ts_tablet_manager.cc:1905] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:08.669793 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd9785888141abaab655f0ba74efb9" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:08.669968 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd9785888141abaab655f0ba74efb9" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.670431 5323 raft_consensus.cc:2463] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:08.670521 5461 raft_consensus.cc:2463] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:08.671552 5111 leader_election.cc:304] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:08.673152 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet d9776df98cb54be38a521852634490e7 (table default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f [id=3bbd1acd709b44899907148df944b20d]) successfully deleted
I20250110 20:57:08.675079 5779 raft_consensus.cc:2798] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:08.675371 5779 raft_consensus.cc:491] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:08.675638 5779 raft_consensus.cc:3054] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:08.681107 5600 tablet_bootstrap.cc:654] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.703822 5594 ts_tablet_manager.cc:1428] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.087s user 0.005s sys 0.000s
I20250110 20:57:08.711628 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 87) took 1540 ms. Trace:
I20250110 20:57:08.711954 5592 rpcz_store.cc:276] 0110 20:57:07.171520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.171669 (+ 149us) service_pool.cc:225] Handling call
0110 20:57:08.457274 (+1285605us) ts_tablet_manager.cc:1145] Deleting tablet 70ac7d62f978433dad004f2908c4bbc4
0110 20:57:08.457312 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:08.560340 (+103028us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:08.662333 (+101993us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:08.711570 (+ 49237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1284964}
I20250110 20:57:08.713099 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 70ac7d62f978433dad004f2908c4bbc4 (table default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]) successfully deleted
I20250110 20:57:08.714699 5594 tablet_bootstrap.cc:492] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:08.714110 5592 tablet_replica.cc:331] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:08.718791 5597 tablet_bootstrap.cc:492] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:08.718951 5592 raft_consensus.cc:2238] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:08.719369 5592 raft_consensus.cc:2267] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.720356 5597 ts_tablet_manager.cc:1397] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.126s user 0.009s sys 0.001s
I20250110 20:57:08.720846 5592 ts_tablet_manager.cc:1905] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:08.723111 5597 raft_consensus.cc:357] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.729270 5600 tablet_bootstrap.cc:492] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.729666 5600 ts_tablet_manager.cc:1397] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.003s
I20250110 20:57:08.731392 5597 raft_consensus.cc:383] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.731806 5600 raft_consensus.cc:357] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.732409 5600 raft_consensus.cc:383] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.732649 5600 raft_consensus.cc:738] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.732333 5597 raft_consensus.cc:738] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.733183 5600 consensus_queue.cc:260] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.735204 5600 ts_tablet_manager.cc:1428] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250110 20:57:08.736635 5600 tablet_bootstrap.cc:492] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.733805 5597 consensus_queue.cc:260] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.745018 5600 tablet_bootstrap.cc:654] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.746832 5594 tablet_bootstrap.cc:654] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.756048 5597 ts_tablet_manager.cc:1428] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.035s user 0.000s sys 0.004s
I20250110 20:57:08.760061 5597 tablet_bootstrap.cc:492] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:08.765724 5779 raft_consensus.cc:513] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.767525 5779 leader_election.cc:290] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:08.769476 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd9785888141abaab655f0ba74efb9" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:08.769644 5461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04fd9785888141abaab655f0ba74efb9" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:08.770004 5323 raft_consensus.cc:3054] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:08.770172 5461 raft_consensus.cc:3054] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:08.773862 5773 ts_tablet_manager.cc:1918] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:08.774226 5773 log.cc:1198] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/8fe93f1a08f3432dba9c6ce76b4c99b0
I20250110 20:57:08.775532 5773 ts_tablet_manager.cc:1939] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:08.779278 5594 tablet_bootstrap.cc:492] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:08.779639 5594 ts_tablet_manager.cc:1397] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.003s
I20250110 20:57:08.780658 5600 tablet_bootstrap.cc:492] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.781910 5600 ts_tablet_manager.cc:1397] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.001s
I20250110 20:57:08.785188 5779 raft_consensus.cc:491] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.785596 5779 raft_consensus.cc:513] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.788535 5779 leader_election.cc:290] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:08.784893 5600 raft_consensus.cc:357] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.790006 5600 raft_consensus.cc:383] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.790581 5600 raft_consensus.cc:738] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.791433 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e43461251413b97721323cd935374" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.791729 5600 consensus_queue.cc:260] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.792117 5464 raft_consensus.cc:2463] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:08.793682 5600 ts_tablet_manager.cc:1428] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250110 20:57:08.794453 5600 tablet_bootstrap.cc:492] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.794199 4898 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_7896d2963d9b4ce78484de7c3e9caf91" } new_table_name: "default.Table_084a1a181cf1428fb3d7af077c77c7bc" modify_external_catalogs: true
I20250110 20:57:08.797950 5809 raft_consensus.cc:491] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.798321 5809 raft_consensus.cc:513] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.801474 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e43461251413b97721323cd935374" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:08.802011 5322 raft_consensus.cc:2463] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:08.803189 4892 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:08.804725 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6eca349514b9fb0fff7cd6230027c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:08.805864 5111 leader_election.cc:304] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:08.807582 5600 tablet_bootstrap.cc:654] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.808929 4896 catalog_manager.cc:5526] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:08.810174 4896 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
W20250110 20:57:08.811287 5257 leader_election.cc:343] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.812245 4893 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:08.815042 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:08.815454 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:08.816054 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:08.816958 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6eca349514b9fb0fff7cd6230027c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:08.818257 5255 leader_election.cc:343] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.818671 5255 leader_election.cc:304] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:08.819365 5713 raft_consensus.cc:2743] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:08.820382 5779 raft_consensus.cc:2798] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:08.821501 5779 raft_consensus.cc:491] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:08.821874 5779 raft_consensus.cc:3054] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:08.831287 4898 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6176c29f01b4ca5888c844c131270f2 consensus state has no leader
W20250110 20:57:08.832366 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet e27e43461251413b97721323cd935374 (table default.Table_b15b46fb1ed048c8b2057b0ab88797cd [id=729d5b2683d04c36ac582ea2965304de], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e27e43461251413b97721323cd935374 consensus state has no leader
I20250110 20:57:08.809783 5809 leader_election.cc:290] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.840314 5594 raft_consensus.cc:357] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.843096 5594 raft_consensus.cc:383] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.844895 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e7ffb063594128ba137f9aea86bf15 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:08.845577 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 70ac7d62f978433dad004f2908c4bbc4 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d67e94c8f9a414499b5e20a5cc0ead3) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:08.846120 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 9155a626d26541ce9d0cf31508f19f15 with delete_type TABLET_DATA_DELETED (replaced by tablet 41de354807004e26bfb147f1d420037e) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:08.846346 5600 tablet_bootstrap.cc:492] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.846724 5600 ts_tablet_manager.cc:1397] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.052s user 0.010s sys 0.000s
I20250110 20:57:08.846714 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet 5793d909c6364047abc85ea5f993c521 with delete_type TABLET_DATA_DELETED (replaced by tablet fcc7b1acf4114bf69d801437855760df) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:08.848778 5600 raft_consensus.cc:357] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.851866 5777 consensus_queue.cc:1035] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:57:08.855445 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet e27e43461251413b97721323cd935374 (table default.Table_b15b46fb1ed048c8b2057b0ab88797cd [id=729d5b2683d04c36ac582ea2965304de]). Expected version 1 got 0
I20250110 20:57:08.854379 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_01d2ed1a594948c4995e0b934e4d2860"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:08.857270 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01d2ed1a594948c4995e0b934e4d2860 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:08.857797 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f1a0fe849b9c44a1b4834b9835cb49f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:08.859534 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1a0fe849b9c44a1b4834b9835cb49f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:08.864832 5773 tablet_replica.cc:331] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:08.849388 5600 raft_consensus.cc:383] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.866076 5600 raft_consensus.cc:738] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.866477 5595 ts_tablet_manager.cc:1918] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:08.866808 5595 log.cc:1198] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:08.867358 5595 ts_tablet_manager.cc:1939] T 31b65488c884452d876ced1c3d1efb9a P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:08.867825 5323 raft_consensus.cc:2463] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:08.868883 5111 leader_election.cc:304] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:08.869622 5777 raft_consensus.cc:2798] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:08.869918 5773 raft_consensus.cc:2238] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:08.870391 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]) successfully deleted
I20250110 20:57:08.870540 5773 pending_rounds.cc:62] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:08.870764 5773 pending_rounds.cc:69] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878605093257216 op_type: NO_OP noop_request { }
W20250110 20:57:08.870991 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 1ac07ac6fc9f477ba7f9accc5533be39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ac07ac6fc9f477ba7f9accc5533be39 already in progress: creating tablet
I20250110 20:57:08.871102 5773 raft_consensus.cc:2883] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:08.871389 5773 raft_consensus.cc:2267] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:08.871517 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d67e94c8f9a414499b5e20a5cc0ead3 already in progress: creating tablet
I20250110 20:57:08.874143 5600 consensus_queue.cc:260] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.877485 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 78538b2e7bcc45c7afaa145bad6e5813
I20250110 20:57:08.878648 4897 catalog_manager.cc:6596] ReplaceTablet: tablet 78538b2e7bcc45c7afaa145bad6e5813 deleted and replaced by tablet 48e53d95ef8645c0bd123138655305ac
I20250110 20:57:08.880649 5600 ts_tablet_manager.cc:1428] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.034s user 0.003s sys 0.002s
I20250110 20:57:08.881028 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 with delete_type TABLET_DATA_DELETED (replaced by tablet 48e53d95ef8645c0bd123138655305ac) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:08.884284 5600 tablet_bootstrap.cc:492] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.889808 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_41125035ac2544fb9a27c3c25133b711"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:08.891268 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41125035ac2544fb9a27c3c25133b711 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:08.892153 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_24d07a4cb95a4deda26177c1bc951dc9" } new_table_name: "default.Table_033612db3a3843f999da6a5beb2b8b47" modify_external_catalogs: true
I20250110 20:57:08.895560 5600 tablet_bootstrap.cc:654] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:08.896067 5809 raft_consensus.cc:491] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.896453 5809 raft_consensus.cc:513] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:08.898129 5809 leader_election.cc:290] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.901463 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_368911cd6dd24875993ac8c44548a12d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:08.902851 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_368911cd6dd24875993ac8c44548a12d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:08.910207 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet f1fafed70adb496d97acfd5fea356292
I20250110 20:57:08.911525 4900 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 139) took 1175 ms. Trace:
I20250110 20:57:08.914551 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet f1fafed70adb496d97acfd5fea356292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:08.918006 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (DEFAULT_TABLE table=default.Table_d40a02bf99094290afe8462ab6a3672f [id=7171d9e7286c4a7caf0423fd3d7d6145]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.919021 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.911823 4900 rpcz_store.cc:276] 0110 20:57:07.735766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.735898 (+ 132us) service_pool.cc:225] Handling call
0110 20:57:07.737152 (+ 1254us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:07.737917 (+ 765us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:07.738054 (+ 137us) catalog_manager.cc:2762] Locking tablets
0110 20:57:07.988613 (+250559us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:08.629566 (+640953us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:08.629782 (+ 216us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:08.910124 (+280342us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:08.911479 (+ 1355us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:07.991960 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:07.992280 (+ 320us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:07.992300 (+ 20us) write_op.cc:435] Acquired schema lock
0110 20:57:07.992312 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:07.992577 (+ 265us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:07.992607 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:07.992627 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:07.992894 (+ 267us) tablet.cc:698] Row locks acquired
0110 20:57:07.992908 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:07.993001 (+ 93us) write_op.cc:270] Start()
0110 20:57:07.993075 (+ 74us) write_op.cc:276] Timestamp: P: 1736542627992904 usec, L: 0
0110 20:57:07.993089 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:07.993537 (+ 448us) log.cc:844] Serialized 762 byte log entry
0110 20:57:08.587287 (+593750us) op_driver.cc:464] REPLICATION: finished
0110 20:57:08.588146 (+ 859us) write_op.cc:301] APPLY: starting
0110 20:57:08.588194 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:08.588278 (+ 84us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:08.588375 (+ 97us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:08.588577 (+ 202us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:08.588706 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:08.588744 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:57:08.588912 (+ 168us) log.cc:844] Serialized 29 byte log entry
0110 20:57:08.625268 (+ 36356us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:08.625339 (+ 71us) write_op.cc:454] Released schema lock
0110 20:57:08.625368 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:08.629979 (+ 4611us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:08.630007 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":707,"apply.run_cpu_time_us":1310,"apply.run_wall_time_us":41797,"num_ops":2,"prepare.queue_time_us":1680,"prepare.run_cpu_time_us":56411,"prepare.run_wall_time_us":241490,"raft.queue_time_us":104171,"raft.run_cpu_time_us":18,"raft.run_wall_time_us":19,"replication_time_us":594114,"thread_start_us":241645,"threads_started":2}]]}
I20250110 20:57:08.921217 5595 tablet_replica.cc:331] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:08.921890 5595 raft_consensus.cc:2238] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:08.922317 5595 raft_consensus.cc:2267] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:08.924140 5595 ts_tablet_manager.cc:1905] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:08.929226 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 31b65488c884452d876ced1c3d1efb9a (table default.Table_d29551acec544d57b14d18fecbf2722e [id=16478fe1cba54c88ba8245c365d5a992]) successfully deleted
I20250110 20:57:08.865895 5592 ts_tablet_manager.cc:1918] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:08.939216 5592 log.cc:1198] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/e38b659f218d42298cb36752b5ca603d
I20250110 20:57:08.939875 5592 ts_tablet_manager.cc:1939] T e38b659f218d42298cb36752b5ca603d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:08.869340 5461 raft_consensus.cc:2463] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:08.885831 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 with delete_type TABLET_DATA_DELETED (replaced by tablet 48e53d95ef8645c0bd123138655305ac) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:08.942471 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 88) took 1763 ms. Trace:
I20250110 20:57:08.942755 5592 rpcz_store.cc:276] 0110 20:57:07.178682 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.178832 (+ 150us) service_pool.cc:225] Handling call
0110 20:57:08.713973 (+1535141us) ts_tablet_manager.cc:1145] Deleting tablet e38b659f218d42298cb36752b5ca603d
0110 20:57:08.714012 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:08.769054 (+ 55042us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:08.865647 (+ 96593us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:08.942410 (+ 76763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1530761}
I20250110 20:57:08.943943 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 90) took 1626 ms. Trace:
I20250110 20:57:08.944657 5592 rpcz_store.cc:276] 0110 20:57:07.317803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.317942 (+ 139us) service_pool.cc:225] Handling call
0110 20:57:08.943633 (+1625691us) ts_tablet_manager.cc:1145] Deleting tablet da24904dece04a6194790054f6d7153a
0110 20:57:08.943668 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:08.943896 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1625192}
I20250110 20:57:08.945493 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet e38b659f218d42298cb36752b5ca603d (table default.Table_4a2a0aff7e744f57bc2c98b76bb0a199 [id=ba089f4f57284f80ac3621a88610fec1]) successfully deleted
I20250110 20:57:08.911600 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet f1fafed70adb496d97acfd5fea356292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:08.916394 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (DEFAULT_TABLE table=default.Table_d40a02bf99094290afe8462ab6a3672f [id=7171d9e7286c4a7caf0423fd3d7d6145]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:08.946512 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet da24904dece04a6194790054f6d7153a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet da24904dece04a6194790054f6d7153a already in progress: creating tablet
I20250110 20:57:08.947096 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.948228 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 637a39e4264b40dd9480dc13e37359d9 (DEFAULT_TABLE table=default.Table_638a8034373e4e3bbfb842beaacdaab8 [id=42aa140491c94323b9a140eea45f28de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.948405 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 637a39e4264b40dd9480dc13e37359d9 (DEFAULT_TABLE table=default.Table_638a8034373e4e3bbfb842beaacdaab8 [id=42aa140491c94323b9a140eea45f28de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.949151 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637a39e4264b40dd9480dc13e37359d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.898824 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da24904dece04a6194790054f6d7153a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:08.954011 5255 leader_election.cc:343] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:08.954993 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 95) took 1572 ms. Trace:
I20250110 20:57:08.955325 5592 rpcz_store.cc:276] 0110 20:57:07.382227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.382766 (+ 539us) service_pool.cc:225] Handling call
0110 20:57:08.954702 (+1571936us) ts_tablet_manager.cc:1145] Deleting tablet 2967c06715e241dcbbe784eed2373704
0110 20:57:08.954738 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:08.954946 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1571446}
I20250110 20:57:08.956785 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 96) took 1407 ms. Trace:
I20250110 20:57:08.957134 5592 rpcz_store.cc:276] 0110 20:57:07.549763 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.549913 (+ 150us) service_pool.cc:225] Handling call
0110 20:57:08.956475 (+1406562us) ts_tablet_manager.cc:1145] Deleting tablet f0b5ba9f38c1493bba01180744f080e1
0110 20:57:08.956510 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:08.956726 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406010}
I20250110 20:57:08.958328 5592 tablet_replica.cc:331] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:08.960426 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 2967c06715e241dcbbe784eed2373704 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2967c06715e241dcbbe784eed2373704 already in progress: creating tablet
W20250110 20:57:08.961043 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet f0b5ba9f38c1493bba01180744f080e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0b5ba9f38c1493bba01180744f080e1 already in progress: creating tablet
I20250110 20:57:08.961742 5809 raft_consensus.cc:491] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.962224 5809 raft_consensus.cc:513] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.964584 5809 leader_election.cc:290] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.967465 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:08.965097 5600 tablet_bootstrap.cc:492] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:08.969732 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637a39e4264b40dd9480dc13e37359d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.969390 5600 ts_tablet_manager.cc:1397] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.003s
I20250110 20:57:08.971930 5809 raft_consensus.cc:491] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.972302 5809 raft_consensus.cc:513] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.972424 5600 raft_consensus.cc:357] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.972997 5600 raft_consensus.cc:383] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:08.974395 5809 leader_election.cc:290] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:08.976966 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]). Expected version 1 got 0
I20250110 20:57:08.959784 5438 tablet_service.cc:1467] Processing CreateTablet for tablet eee23ab7ecce465ca1153d22c40bf935 (DEFAULT_TABLE table=default.Table_35bec2f295574ca1bf89eaea0cd3942c [id=a02045f76da84eea801f9a70059d4d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:08.968161 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5ba9f38c1493bba01180744f080e1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.975876 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310cf52aa8c4930a7821117a46dc958" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:08.973220 5600 raft_consensus.cc:738] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:08.983317 5600 consensus_queue.cc:260] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.985520 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee23ab7ecce465ca1153d22c40bf935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:08.987931 5600 ts_tablet_manager.cc:1428] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20250110 20:57:08.989913 5600 tablet_bootstrap.cc:492] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:08.991856 5713 raft_consensus.cc:491] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:08.992254 5713 raft_consensus.cc:513] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:08.993965 5713 leader_election.cc:290] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:09.000245 5255 leader_election.cc:343] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:09.000948 5255 leader_election.cc:343] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.002106 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:09.002681 5600 tablet_bootstrap.cc:654] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:09.013878 5255 leader_election.cc:343] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.015923 5592 raft_consensus.cc:2238] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.016428 5592 raft_consensus.cc:2267] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.018025 5592 ts_tablet_manager.cc:1905] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.025120 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 with delete_type TABLET_DATA_DELETED (replaced by tablet 48e53d95ef8645c0bd123138655305ac) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:09.026868 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:09.029948 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet 2967c06715e241dcbbe784eed2373704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:09.030979 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:09.032485 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 0af278622ef6432fb0891cb06bc6cd6d (DEFAULT_TABLE table=default.Table_1ae0e1ebb3fa451baeb54dfb50eef3a5 [id=0df25099e0f4482a8881a258a954312a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.033921 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet da24904dece04a6194790054f6d7153a with delete_type TABLET_DATA_DELETED (replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:09.034492 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:09.034960 5296 tablet_service.cc:1467] Processing CreateTablet for tablet fcc7b1acf4114bf69d801437855760df (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.037142 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (DEFAULT_TABLE table=default.Table_a809544a8ed648619ae05e19bdd40599 [id=79b8eb01b2344a4897e1bc773e7d7935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.046121 5292 tablet_service.cc:1467] Processing CreateTablet for tablet eee23ab7ecce465ca1153d22c40bf935 (DEFAULT_TABLE table=default.Table_35bec2f295574ca1bf89eaea0cd3942c [id=a02045f76da84eea801f9a70059d4d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.048878 4891 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_638a8034373e4e3bbfb842beaacdaab8" } modify_external_catalogs: true
I20250110 20:57:09.049523 4891 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_638a8034373e4e3bbfb842beaacdaab8" } modify_external_catalogs: true
I20250110 20:57:09.057189 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 0af278622ef6432fb0891cb06bc6cd6d (DEFAULT_TABLE table=default.Table_1ae0e1ebb3fa451baeb54dfb50eef3a5 [id=0df25099e0f4482a8881a258a954312a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.058274 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af278622ef6432fb0891cb06bc6cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.066716 5434 tablet_service.cc:1467] Processing CreateTablet for tablet fcc7b1acf4114bf69d801437855760df (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.067687 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc7b1acf4114bf69d801437855760df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.070279 5595 ts_tablet_manager.cc:1918] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:09.070364 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]). Expected version 1 got 0
I20250110 20:57:09.070602 5595 log.cc:1198] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:09.071192 5595 ts_tablet_manager.cc:1939] T 3a520230d9fd4a74914e46f784dcf747 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:09.071857 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee23ab7ecce465ca1153d22c40bf935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.073877 5437 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (DEFAULT_TABLE table=default.Table_a809544a8ed648619ae05e19bdd40599 [id=79b8eb01b2344a4897e1bc773e7d7935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.074838 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5c46de95bc4227bffc5e8425b8cc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.077666 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc7b1acf4114bf69d801437855760df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.077865 4898 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b15b46fb1ed048c8b2057b0ab88797cd" } new_table_name: "default.Table_ead31c28470a4d3384b767df0ecce1ef" modify_external_catalogs: true
I20250110 20:57:09.078929 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5c46de95bc4227bffc5e8425b8cc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.080116 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af278622ef6432fb0891cb06bc6cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.089654 5595 tablet_replica.cc:331] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:09.090420 5595 raft_consensus.cc:2238] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.090911 5595 raft_consensus.cc:2267] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.092588 5595 ts_tablet_manager.cc:1905] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.093140 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 3a520230d9fd4a74914e46f784dcf747 (table default.Table_8df7bbf6722249b2be68514a607459b9 [id=64cfe285ff2041778f15cf7e6020e9cb]) successfully deleted
I20250110 20:57:09.106599 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e17e3d34a10d4c05b4ae644cd35c9263"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:09.108090 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e17e3d34a10d4c05b4ae644cd35c9263 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:09.110821 5600 tablet_bootstrap.cc:492] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.111248 5600 ts_tablet_manager.cc:1397] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.004s
I20250110 20:57:09.113315 5600 raft_consensus.cc:357] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.114028 5600 raft_consensus.cc:383] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.116585 5600 raft_consensus.cc:738] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.122315 4894 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_792d0af8e9d04ad28d950c7656211372" } modify_external_catalogs: true
I20250110 20:57:09.122840 4894 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_792d0af8e9d04ad28d950c7656211372" } modify_external_catalogs: true
I20250110 20:57:09.139137 5777 raft_consensus.cc:695] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:09.140825 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (DEFAULT_TABLE table=default.Table_d40a02bf99094290afe8462ab6a3672f [id=7171d9e7286c4a7caf0423fd3d7d6145]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.140895 5777 consensus_queue.cc:237] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.141927 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.117295 5600 consensus_queue.cc:260] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.144645 5600 ts_tablet_manager.cc:1428] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.033s user 0.004s sys 0.000s
I20250110 20:57:09.154237 5773 ts_tablet_manager.cc:1905] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.161093 5600 tablet_bootstrap.cc:492] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.139447 5779 raft_consensus.cc:513] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:09.140074 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 115457us
I20250110 20:57:09.172679 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da24904dece04a6194790054f6d7153a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:09.173754 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5ba9f38c1493bba01180744f080e1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:09.174662 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310cf52aa8c4930a7821117a46dc958" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:09.175567 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:09.180712 5809 raft_consensus.cc:491] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.181190 5809 raft_consensus.cc:513] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.183640 5779 leader_election.cc:290] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:09.191381 5809 leader_election.cc:290] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:09.188436 5600 tablet_bootstrap.cc:654] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.197083 4892 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_e470babc3c014ea5bba249532d50babf [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 1 got 0
W20250110 20:57:09.203060 5257 leader_election.cc:343] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.215399 5594 raft_consensus.cc:738] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.216598 5257 leader_election.cc:304] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:09.217480 5809 raft_consensus.cc:2743] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:09.218514 5257 leader_election.cc:343] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.218938 5257 leader_election.cc:304] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:09.219169 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae5956b1aa17449497f9695a111ebb66 consensus state has no leader
I20250110 20:57:09.219564 5809 raft_consensus.cc:2743] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.220520 5600 tablet_bootstrap.cc:492] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.220961 5600 ts_tablet_manager.cc:1397] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.000s
I20250110 20:57:09.186908 5597 tablet_bootstrap.cc:654] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.223207 5600 raft_consensus.cc:357] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.226210 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet f1fafed70adb496d97acfd5fea356292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:09.227187 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 637a39e4264b40dd9480dc13e37359d9 (DEFAULT_TABLE table=default.Table_638a8034373e4e3bbfb842beaacdaab8 [id=42aa140491c94323b9a140eea45f28de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.228101 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 637a39e4264b40dd9480dc13e37359d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.230134 5144 tablet_service.cc:1467] Processing CreateTablet for tablet eee23ab7ecce465ca1153d22c40bf935 (DEFAULT_TABLE table=default.Table_35bec2f295574ca1bf89eaea0cd3942c [id=a02045f76da84eea801f9a70059d4d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.230619 5600 raft_consensus.cc:383] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.230877 5600 raft_consensus.cc:738] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.231160 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eee23ab7ecce465ca1153d22c40bf935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.231436 5600 consensus_queue.cc:260] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.234108 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 0af278622ef6432fb0891cb06bc6cd6d (DEFAULT_TABLE table=default.Table_1ae0e1ebb3fa451baeb54dfb50eef3a5 [id=0df25099e0f4482a8881a258a954312a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.235100 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0af278622ef6432fb0891cb06bc6cd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.238178 5600 ts_tablet_manager.cc:1428] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20250110 20:57:09.238147 5155 tablet_service.cc:1467] Processing CreateTablet for tablet fcc7b1acf4114bf69d801437855760df (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.239030 5600 tablet_bootstrap.cc:492] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.239077 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc7b1acf4114bf69d801437855760df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.242714 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (DEFAULT_TABLE table=default.Table_a809544a8ed648619ae05e19bdd40599 [id=79b8eb01b2344a4897e1bc773e7d7935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.243573 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d5c46de95bc4227bffc5e8425b8cc95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.245258 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet 2d634add82994f74960cd2b8373a4dc6 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:09.246692 5809 raft_consensus.cc:491] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.247195 5809 raft_consensus.cc:513] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.249503 5809 leader_election.cc:290] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:09.251497 5257 leader_election.cc:343] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.251856 5257 leader_election.cc:304] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:09.252437 5809 raft_consensus.cc:2743] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.256621 5597 tablet_bootstrap.cc:492] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:09.259624 5597 ts_tablet_manager.cc:1397] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.500s user 0.007s sys 0.002s
W20250110 20:57:09.263348 5257 leader_election.cc:343] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.263618 4893 master_service.cc:946] ReplaceTablet: received request to replace tablet 2967c06715e241dcbbe784eed2373704 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:09.263695 5257 leader_election.cc:304] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:09.264276 5809 raft_consensus.cc:2743] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.266410 5597 raft_consensus.cc:357] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.271950 5600 tablet_bootstrap.cc:654] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.266937 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2967c06715e241dcbbe784eed2373704" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:09.273528 5597 raft_consensus.cc:383] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.273545 5595 ts_tablet_manager.cc:1918] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:09.273844 5595 log.cc:1198] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/961fef96e9124aafb1c51e287f603b64
I20250110 20:57:09.274391 5595 ts_tablet_manager.cc:1939] T 961fef96e9124aafb1c51e287f603b64 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:09.273806 5597 raft_consensus.cc:738] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.277944 5595 tablet_replica.cc:331] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:09.278155 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 961fef96e9124aafb1c51e287f603b64 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]) successfully deleted
I20250110 20:57:09.278590 5595 raft_consensus.cc:2238] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.278964 5595 raft_consensus.cc:2267] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.280439 5595 ts_tablet_manager.cc:1905] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.278949 5597 consensus_queue.cc:260] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:09.283663 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.254:34617, user_credentials={real_user=slave}} blocked reactor thread for 67142.4us
I20250110 20:57:09.285297 5597 ts_tablet_manager.cc:1428] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.022s user 0.003s sys 0.001s
I20250110 20:57:09.290177 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:09.293125 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e43461251413b97721323cd935374" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:09.293660 5322 raft_consensus.cc:3054] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:09.296926 5809 raft_consensus.cc:491] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.298233 5809 raft_consensus.cc:513] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.300769 5809 leader_election.cc:290] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:09.302011 5257 leader_election.cc:343] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.306639 5597 tablet_bootstrap.cc:492] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:09.309741 5257 leader_election.cc:343] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.313357 5833 consensus_queue.cc:1035] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:09.313732 5600 tablet_bootstrap.cc:492] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.314090 5600 ts_tablet_manager.cc:1397] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.007s
I20250110 20:57:09.316111 5600 raft_consensus.cc:357] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.316735 5600 raft_consensus.cc:383] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.316970 5600 raft_consensus.cc:738] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.320045 5597 tablet_bootstrap.cc:654] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.321985 5600 consensus_queue.cc:260] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.325343 5600 ts_tablet_manager.cc:1428] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.011s user 0.004s sys 0.001s
I20250110 20:57:09.326200 5600 tablet_bootstrap.cc:492] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.328724 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_e470babc3c014ea5bba249532d50babf [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 1 got 0
I20250110 20:57:09.332015 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:09.334512 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_1498d22e348d4599a5514859d6371910"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:09.336261 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1498d22e348d4599a5514859d6371910 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:09.324862 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:09.346691 5464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2967c06715e241dcbbe784eed2373704" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:09.352147 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:09.349311 5257 leader_election.cc:343] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.354076 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:09.356823 5255 leader_election.cc:343] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.357183 5255 leader_election.cc:304] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:09.358135 5255 leader_election.cc:343] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.358476 5255 leader_election.cc:304] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:09.359647 5809 raft_consensus.cc:2743] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.360168 5809 raft_consensus.cc:2743] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.376673 5322 raft_consensus.cc:2463] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:09.355928 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e27e43461251413b97721323cd935374" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:09.383160 5458 raft_consensus.cc:3054] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:09.384852 5600 tablet_bootstrap.cc:654] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.391575 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:09.394475 5592 ts_tablet_manager.cc:1918] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:09.394865 5592 log.cc:1198] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/a7e7ffb063594128ba137f9aea86bf15
I20250110 20:57:09.395553 5592 ts_tablet_manager.cc:1939] T a7e7ffb063594128ba137f9aea86bf15 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:09.401262 5111 leader_election.cc:304] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:09.402021 5833 raft_consensus.cc:2798] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:09.402419 5833 raft_consensus.cc:695] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:09.403154 5833 consensus_queue.cc:237] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.415119 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:09.423591 4899 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b4ab0cd248e44e67ab389fe94e4a1bab" } new_table_name: "default.Table_7b913fb45b9d478ba496dbf9d6703f30" modify_external_catalogs: true
I20250110 20:57:09.432369 5600 tablet_bootstrap.cc:492] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.432798 5600 ts_tablet_manager.cc:1397] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.016s
I20250110 20:57:09.439429 5600 raft_consensus.cc:357] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.440061 5600 raft_consensus.cc:383] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.440299 5600 raft_consensus.cc:738] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.440861 5600 consensus_queue.cc:260] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.444180 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a0178b82e1d8457eb7f165640e222c72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:09.446012 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0178b82e1d8457eb7f165640e222c72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:09.216035 5594 consensus_queue.cc:260] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.453976 5600 ts_tablet_manager.cc:1428] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.021s user 0.004s sys 0.003s
I20250110 20:57:09.454756 5600 tablet_bootstrap.cc:492] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:09.458377 5112 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.171.254:34617, user_credentials={real_user=slave}} blocked reactor thread for 137428us
I20250110 20:57:09.469619 5600 tablet_bootstrap.cc:654] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.479074 5595 ts_tablet_manager.cc:1918] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:09.479411 5595 log.cc:1198] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:09.479940 5595 ts_tablet_manager.cc:1939] T 1ac07ac6fc9f477ba7f9accc5533be39 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:09.480630 5458 raft_consensus.cc:2463] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
W20250110 20:57:09.481797 5255 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 123639us
W20250110 20:57:09.482802 5255 leader_election.cc:343] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.483145 5255 leader_election.cc:304] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:09.484145 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 97) took 1893 ms. Trace:
I20250110 20:57:09.484268 5713 raft_consensus.cc:2743] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.484460 5592 rpcz_store.cc:276] 0110 20:57:07.590626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.590757 (+ 131us) service_pool.cc:225] Handling call
0110 20:57:08.958202 (+1367445us) ts_tablet_manager.cc:1145] Deleting tablet a7e7ffb063594128ba137f9aea86bf15
0110 20:57:08.958237 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:09.281584 (+323347us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:09.394279 (+112695us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:09.484090 (+ 89811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1366927}
I20250110 20:57:09.485949 5595 tablet_replica.cc:331] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:09.485889 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet a7e7ffb063594128ba137f9aea86bf15 (table default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]) successfully deleted
I20250110 20:57:09.486267 5600 tablet_bootstrap.cc:492] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.486611 5595 raft_consensus.cc:2238] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.486837 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 98) took 1853 ms. Trace:
I20250110 20:57:09.487031 5595 raft_consensus.cc:2267] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.487112 5592 rpcz_store.cc:276] 0110 20:57:07.633180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:07.633300 (+ 120us) service_pool.cc:225] Handling call
0110 20:57:09.485388 (+1852088us) ts_tablet_manager.cc:1145] Deleting tablet 5793d909c6364047abc85ea5f993c521
0110 20:57:09.486515 (+ 1127us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:09.486788 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1851475}
I20250110 20:57:09.487962 5592 tablet_replica.cc:331] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:09.486622 5600 ts_tablet_manager.cc:1397] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.003s
W20250110 20:57:09.489692 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 5793d909c6364047abc85ea5f993c521 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5793d909c6364047abc85ea5f993c521 already in progress: creating tablet
I20250110 20:57:09.490340 5592 raft_consensus.cc:2238] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.491268 5592 pending_rounds.cc:62] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:09.491544 5592 pending_rounds.cc:69] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878604151046144 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "8fe93f1a08f3432dba9c6ce76b4c99b0" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_b115c55a632e444795c1dfc779b62d48" dest_uuid: "ac9003be04f340f3886b5625cc928eb0" new_extra_config { } }
I20250110 20:57:09.492127 5595 ts_tablet_manager.cc:1905] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.492792 5592 raft_consensus.cc:2267] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.494426 5592 ts_tablet_manager.cc:1905] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.491818 5600 raft_consensus.cc:357] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.498320 5600 raft_consensus.cc:383] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.498585 5600 raft_consensus.cc:738] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:09.499573 5441 consensus_meta.cc:287] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 0.519s user 0.004s sys 0.003s
I20250110 20:57:09.500700 5600 consensus_queue.cc:260] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.505071 5600 ts_tablet_manager.cc:1428] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.016s user 0.003s sys 0.002s
I20250110 20:57:09.508772 5600 tablet_bootstrap.cc:492] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.515314 5833 raft_consensus.cc:491] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.515687 5833 raft_consensus.cc:513] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.518479 5833 leader_election.cc:290] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:09.519124 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477b09061a4a4f278720168872a83258" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:09.520188 5459 raft_consensus.cc:2463] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:09.521129 5110 leader_election.cc:304] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:09.525579 5600 tablet_bootstrap.cc:654] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:09.525727 5111 leader_election.cc:343] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 477b09061a4a4f278720168872a83258
I20250110 20:57:09.527813 5594 ts_tablet_manager.cc:1428] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.748s user 0.006s sys 0.001s
W20250110 20:57:09.528097 5594 ts_tablet_manager.cc:1460] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Tablet startup took 748ms
W20250110 20:57:09.541078 5255 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 58297.5us
I20250110 20:57:09.542311 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 1ac07ac6fc9f477ba7f9accc5533be39 (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]) successfully deleted
I20250110 20:57:09.543584 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:09.547622 5600 tablet_bootstrap.cc:492] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.548005 5600 ts_tablet_manager.cc:1397] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.004s
I20250110 20:57:09.550124 5600 raft_consensus.cc:357] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.550710 5600 raft_consensus.cc:383] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.550982 5600 raft_consensus.cc:738] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.555555 5597 tablet_bootstrap.cc:492] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
W20250110 20:57:09.555428 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.252:37233, user_credentials={real_user=slave}} blocked reactor thread for 238105us
I20250110 20:57:09.555949 5597 ts_tablet_manager.cc:1397] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.249s user 0.020s sys 0.045s
I20250110 20:57:09.558141 5597 raft_consensus.cc:357] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.558732 5597 raft_consensus.cc:383] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.558966 5597 raft_consensus.cc:738] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.554255 5600 consensus_queue.cc:260] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.562428 5600 ts_tablet_manager.cc:1428] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.014s user 0.004s sys 0.001s
I20250110 20:57:09.562649 4891 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 637a39e4264b40dd9480dc13e37359d9
I20250110 20:57:09.563164 5600 tablet_bootstrap.cc:492] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.566435 5597 consensus_queue.cc:260] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.568663 5827 raft_consensus.cc:2798] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:09.568977 5827 raft_consensus.cc:491] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:09.569250 5827 raft_consensus.cc:3054] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:09.576587 5597 ts_tablet_manager.cc:1428] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20250110 20:57:09.579243 5597 tablet_bootstrap.cc:492] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:09.579910 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 637a39e4264b40dd9480dc13e37359d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:09.582453 5846 raft_consensus.cc:491] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.582506 5827 raft_consensus.cc:513] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.582898 5846 raft_consensus.cc:513] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.584619 5846 leader_election.cc:290] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:09.585922 5827 leader_election.cc:290] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:09.586560 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "477b09061a4a4f278720168872a83258" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:09.587205 5459 raft_consensus.cc:3054] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:09.590919 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:09.593748 5257 leader_election.cc:343] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:09.595167 5111 leader_election.cc:343] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 477b09061a4a4f278720168872a83258
W20250110 20:57:09.528326 5594 ts_tablet_manager.cc:1462] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trace:
0110 20:57:04.744254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:04.767033 (+ 22779us) service_pool.cc:225] Handling call
0110 20:57:04.814802 (+ 47769us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b5800001898. stack: 00007f2677942853 00007f2677942750 00007f26774b7584 00000000004fcd24 00000000004fccb9 00007f26777b82d1 00007f2677937291 00007f2677937142 00007f26805a0cd9 00007f267f82e750 00007f267f910d91 00007f267f8ee243 00007f267f89f859 00007f267df87c64 00007f267df87bf2 00007f267df87b21
0110 20:57:04.814930 (+ 128us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:04.815012 (+ 82us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:04.924029 (+109017us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:05.020059 (+ 96030us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:05.020435 (+ 376us) inbound_call.cc:173] Queueing success response
0110 20:57:08.714450 (+3694015us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0110 20:57:08.779872 (+ 65422us) ts_tablet_manager.cc:1429] Starting tablet replica
0110 20:57:08.780304 (+ 432us) tablet_replica.cc:260] Starting instrumentation
0110 20:57:08.780712 (+ 408us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":21969,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":124,"spinlock_wait_cycles":5822208,"tablet-open.queue_time_us":3694089,"thread_start_us":75263,"threads_started":1}
I20250110 20:57:09.601234 5597 tablet_bootstrap.cc:654] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.600234 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:09.602239 5600 tablet_bootstrap.cc:654] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:09.604005 4789 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 47036.9us
I20250110 20:57:09.608242 5594 tablet_bootstrap.cc:492] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:09.612871 5600 tablet_bootstrap.cc:492] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.614543 5600 ts_tablet_manager.cc:1397] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.008s
W20250110 20:57:09.615406 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.254:34617, user_credentials={real_user=slave}} blocked reactor thread for 57576.3us
I20250110 20:57:09.616524 5600 raft_consensus.cc:357] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.617133 5600 raft_consensus.cc:383] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.620864 5597 tablet_bootstrap.cc:492] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:09.621228 5597 ts_tablet_manager.cc:1397] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
I20250110 20:57:09.623248 5597 raft_consensus.cc:357] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.623867 5597 raft_consensus.cc:383] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.624109 5597 raft_consensus.cc:738] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.624620 5597 consensus_queue.cc:260] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.627609 5852 raft_consensus.cc:491] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.617352 5600 raft_consensus.cc:738] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.629825 5600 consensus_queue.cc:260] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.633493 5600 ts_tablet_manager.cc:1428] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250110 20:57:09.634218 5600 tablet_bootstrap.cc:492] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.639521 5597 ts_tablet_manager.cc:1428] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20250110 20:57:09.640333 5597 tablet_bootstrap.cc:492] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:09.650825 5600 tablet_bootstrap.cc:654] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.660972 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 637a39e4264b40dd9480dc13e37359d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:09.663363 5597 tablet_bootstrap.cc:654] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:09.664441 5255 leader_election.cc:343] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.664914 5255 leader_election.cc:304] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:09.665644 5846 raft_consensus.cc:2743] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.670150 5600 tablet_bootstrap.cc:492] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.670509 5600 ts_tablet_manager.cc:1397] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.005s
I20250110 20:57:09.672668 4900 master_service.cc:946] ReplaceTablet: received request to replace tablet c6914e258314423c859eb31e85853274 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:09.677079 5597 tablet_bootstrap.cc:492] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:09.677441 5597 ts_tablet_manager.cc:1397] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.004s
I20250110 20:57:09.680922 5592 ts_tablet_manager.cc:1918] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:09.681826 5773 ts_tablet_manager.cc:1918] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:09.681563 5597 raft_consensus.cc:357] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.682157 5773 log.cc:1198] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/a7e7ffb063594128ba137f9aea86bf15
I20250110 20:57:09.682194 5597 raft_consensus.cc:383] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.682464 5597 raft_consensus.cc:738] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.682897 5773 ts_tablet_manager.cc:1939] T a7e7ffb063594128ba137f9aea86bf15 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:09.683284 4897 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:09.682997 5597 consensus_queue.cc:260] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.681282 5592 log.cc:1198] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/8fe93f1a08f3432dba9c6ce76b4c99b0
I20250110 20:57:09.684128 5592 ts_tablet_manager.cc:1939] T 8fe93f1a08f3432dba9c6ce76b4c99b0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:09.673070 5600 raft_consensus.cc:357] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.684612 5600 raft_consensus.cc:383] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.684880 5600 raft_consensus.cc:738] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.687518 5600 consensus_queue.cc:260] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.688587 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 637a39e4264b40dd9480dc13e37359d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:09.695755 5600 ts_tablet_manager.cc:1428] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20250110 20:57:09.696525 5600 tablet_bootstrap.cc:492] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.705443 5597 ts_tablet_manager.cc:1428] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.028s user 0.003s sys 0.002s
I20250110 20:57:09.710848 5597 tablet_bootstrap.cc:492] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:09.712234 5773 tablet_replica.cc:331] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:09.712945 5773 raft_consensus.cc:2238] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.713351 5773 raft_consensus.cc:2267] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.714985 5773 ts_tablet_manager.cc:1905] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.716017 5597 tablet_bootstrap.cc:654] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.722304 5600 tablet_bootstrap.cc:654] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.722424 5833 raft_consensus.cc:491] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.722821 5833 raft_consensus.cc:513] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.724524 5833 leader_election.cc:290] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:09.725939 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78538b2e7bcc45c7afaa145bad6e5813" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:09.726495 5323 raft_consensus.cc:2463] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:09.727555 5111 leader_election.cc:304] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:09.728405 5833 raft_consensus.cc:2798] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:09.728716 5833 raft_consensus.cc:491] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:09.729008 5833 raft_consensus.cc:3054] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:09.733558 5600 tablet_bootstrap.cc:492] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:09.735229 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 99) took 1758 ms. Trace:
I20250110 20:57:09.735540 5592 rpcz_store.cc:276] 0110 20:57:07.977330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.022702 (+ 45372us) service_pool.cc:225] Handling call
0110 20:57:09.487838 (+1465136us) ts_tablet_manager.cc:1145] Deleting tablet 8fe93f1a08f3432dba9c6ce76b4c99b0
0110 20:57:09.487871 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:09.580643 (+ 92772us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:09.680706 (+100063us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:09.735167 (+ 54461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1464675}
I20250110 20:57:09.736536 5592 tablet_replica.cc:331] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:09.739009 5597 tablet_bootstrap.cc:492] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:09.738778 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78538b2e7bcc45c7afaa145bad6e5813" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:09.739382 5597 ts_tablet_manager.cc:1397] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20250110 20:57:09.734102 5459 raft_consensus.cc:2463] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:09.733940 5600 ts_tablet_manager.cc:1397] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.003s
I20250110 20:57:09.740217 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 (table default.Table_b115c55a632e444795c1dfc779b62d48 [id=07ca97491a46431291928447b4317221]) successfully deleted
W20250110 20:57:09.741441 5110 leader_election.cc:343] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.742274 5110 leader_election.cc:304] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:09.742327 5600 raft_consensus.cc:357] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.742937 5600 raft_consensus.cc:383] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.742972 5827 raft_consensus.cc:2798] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:09.743192 5600 raft_consensus.cc:738] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.743372 5827 raft_consensus.cc:695] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:09.743731 5592 raft_consensus.cc:2238] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.743727 5600 consensus_queue.cc:260] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.744048 5827 consensus_queue.cc:237] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.745555 5592 raft_consensus.cc:2267] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.747115 5592 ts_tablet_manager.cc:1905] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.747112 5597 raft_consensus.cc:357] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.747745 5597 raft_consensus.cc:383] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.748044 5597 raft_consensus.cc:738] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.628043 5852 raft_consensus.cc:513] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:09.748596 5597 consensus_queue.cc:260] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:09.754365 5398 leader_election.cc:343] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 5793d909c6364047abc85ea5f993c521
I20250110 20:57:09.756844 5852 leader_election.cc:290] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:09.758297 5600 ts_tablet_manager.cc:1428] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20250110 20:57:09.759498 5600 tablet_bootstrap.cc:492] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:09.761852 5597 ts_tablet_manager.cc:1428] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20250110 20:57:09.762650 5597 tablet_bootstrap.cc:492] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:09.764528 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 5793d909c6364047abc85ea5f993c521 with delete_type TABLET_DATA_DELETED (replaced by tablet fcc7b1acf4114bf69d801437855760df) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:09.771696 5597 tablet_bootstrap.cc:654] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:09.772480 5399 leader_election.cc:343] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: 5793d909c6364047abc85ea5f993c521
I20250110 20:57:09.773607 5399 leader_election.cc:304] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:09.778734 5852 raft_consensus.cc:2743] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:09.781505 5600 tablet_bootstrap.cc:654] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:09.781811 5112 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.4.171.254:34617, user_credentials={real_user=slave}} blocked reactor thread for 78582.4us
I20250110 20:57:09.783890 5595 ts_tablet_manager.cc:1918] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:09.784231 5595 log.cc:1198] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:09.784766 5595 ts_tablet_manager.cc:1939] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:09.788074 5833 raft_consensus.cc:513] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:09.789040 5595 tablet_replica.cc:331] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:09.789731 5595 raft_consensus.cc:2238] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:09.790131 5595 raft_consensus.cc:2267] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:09.791705 5595 ts_tablet_manager.cc:1905] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:09.791893 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78538b2e7bcc45c7afaa145bad6e5813" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:09.792133 5833 leader_election.cc:290] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:09.793339 5111 leader_election.cc:343] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.793723 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78538b2e7bcc45c7afaa145bad6e5813" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
W20250110 20:57:09.794936 5110 leader_election.cc:343] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:09.795351 5110 leader_election.cc:304] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:09.797392 5833 raft_consensus.cc:2743] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250110 20:57:09.798846 4899 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2644e9180a9548c08cf9e7c4bebb8189 has no consensus state
I20250110 20:57:09.810344 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 8d67e94c8f9a414499b5e20a5cc0ead3 (table default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]) successfully deleted
W20250110 20:57:09.833426 4886 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2644e9180a9548c08cf9e7c4bebb8189 has no consensus state
I20250110 20:57:09.838414 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet a7e7ffb063594128ba137f9aea86bf15 (table default.Table_0fb62c9c09f741c7beea780971c66f3a [id=2350ca295780453892667474d5d25447]) successfully deleted
I20250110 20:57:09.841967 5594 tablet_bootstrap.cc:654] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.849349 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_1c8384b6009d43a89936a5f543415675"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:09.851426 5597 tablet_bootstrap.cc:492] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
W20250110 20:57:09.851518 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c8384b6009d43a89936a5f543415675 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:09.851819 5597 ts_tablet_manager.cc:1397] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.019s
I20250110 20:57:09.854781 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_c3b44e7a68ac47a1ad0f7c5496130418" } modify_external_catalogs: true
I20250110 20:57:09.855348 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_c3b44e7a68ac47a1ad0f7c5496130418" } modify_external_catalogs: true
I20250110 20:57:09.857909 5597 raft_consensus.cc:357] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.858431 5594 tablet_bootstrap.cc:492] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:09.858527 5597 raft_consensus.cc:383] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.858791 5594 ts_tablet_manager.cc:1397] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.251s user 0.008s sys 0.001s
I20250110 20:57:09.858759 5597 raft_consensus.cc:738] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.859292 5597 consensus_queue.cc:260] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.861172 5597 ts_tablet_manager.cc:1428] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250110 20:57:09.860930 5594 raft_consensus.cc:357] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.865078 5594 raft_consensus.cc:383] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:09.865661 5594 raft_consensus.cc:738] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:09.866591 5594 consensus_queue.cc:260] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:09.869771 5594 ts_tablet_manager.cc:1428] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20250110 20:57:09.873236 5594 tablet_bootstrap.cc:492] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:09.878310 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_338aa5da81e146be96e0f8cc5dcae996"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:09.879825 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_338aa5da81e146be96e0f8cc5dcae996 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:09.883327 5594 tablet_bootstrap.cc:654] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.895081 5773 ts_tablet_manager.cc:1918] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:09.895491 5773 log.cc:1198] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/70ac7d62f978433dad004f2908c4bbc4
I20250110 20:57:09.896193 5773 ts_tablet_manager.cc:1939] T 70ac7d62f978433dad004f2908c4bbc4 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:09.895541 5597 tablet_bootstrap.cc:492] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:09.908391 5597 tablet_bootstrap.cc:654] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:09.924032 5833 raft_consensus.cc:491] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:09.924459 5833 raft_consensus.cc:513] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:09.930248 5111 leader_election.cc:343] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 961fef96e9124aafb1c51e287f603b64
I20250110 20:57:09.927003 5833 leader_election.cc:290] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:09.982466 4892 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:09.983855 4892 catalog_manager.cc:5526] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:09.985970 4892 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:09.987287 4892 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet e27e43461251413b97721323cd935374 (table default.Table_b15b46fb1ed048c8b2057b0ab88797cd [id=729d5b2683d04c36ac582ea2965304de]). Expected version 1 got 0
W20250110 20:57:09.988444 4898 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e27e43461251413b97721323cd935374 consensus state has no leader
I20250110 20:57:09.991928 5595 ts_tablet_manager.cc:1918] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:09.992280 5595 log.cc:1198] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/78538b2e7bcc45c7afaa145bad6e5813
I20250110 20:57:09.992075 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 432078ac2fe347e1a55834ca0f77531a (DEFAULT_TABLE table=default.Table_5f0518ca447a401eb8fe8655ecdc9a55 [id=ea236ef3b89c437a84ee6ec4a1cecf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.992851 5595 ts_tablet_manager.cc:1939] T 78538b2e7bcc45c7afaa145bad6e5813 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:09.993149 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432078ac2fe347e1a55834ca0f77531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.998731 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 2644e9180a9548c08cf9e7c4bebb8189 (DEFAULT_TABLE table=default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:09.998711 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_16d52a446dab43b6929b4d8fcce5e5f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:09.999593 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2644e9180a9548c08cf9e7c4bebb8189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:09.999425 5301 tablet_service.cc:1467] Processing CreateTablet for tablet c44ca47c747544b6afec83b399dcdcb5 (DEFAULT_TABLE table=default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:10.000082 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16d52a446dab43b6929b4d8fcce5e5f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:10.000355 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c44ca47c747544b6afec83b399dcdcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.000901 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 57ef3773a87e4a23a19ac1e5f603d2ce (DEFAULT_TABLE table=default.Table_61b3b8a681344e25acecd34b7455bb08 [id=3392c523ded14b0d899f563eaf1ed148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.001842 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ef3773a87e4a23a19ac1e5f603d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.002091 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 371fb7408f0149c8b00659e09ebaf48a (DEFAULT_TABLE table=default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.002982 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371fb7408f0149c8b00659e09ebaf48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.004729 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e470babc3c014ea5bba249532d50babf" } new_table_name: "default.Table_d1c3754a45fb4d75b94dffa666cfc9a8" modify_external_catalogs: true
I20250110 20:57:10.006368 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a2199a41b9e74b16ab3d1b90b34ddad7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:10.007727 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2199a41b9e74b16ab3d1b90b34ddad7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:10.008733 5294 tablet_service.cc:1467] Processing CreateTablet for tablet e508061e93cc430cb88f6d10d9684f00 (DEFAULT_TABLE table=default.Table_01d2ed1a594948c4995e0b934e4d2860 [id=e9f106d1b737444baaea6e97a8a3884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.009802 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e508061e93cc430cb88f6d10d9684f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.021555 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 48e53d95ef8645c0bd123138655305ac (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.022434 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e53d95ef8645c0bd123138655305ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.026741 5434 tablet_service.cc:1467] Processing CreateTablet for tablet c44ca47c747544b6afec83b399dcdcb5 (DEFAULT_TABLE table=default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.028581 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c44ca47c747544b6afec83b399dcdcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.035391 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 2644e9180a9548c08cf9e7c4bebb8189 (DEFAULT_TABLE table=default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.036442 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2644e9180a9548c08cf9e7c4bebb8189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.036644 5145 tablet_service.cc:1467] Processing CreateTablet for tablet e508061e93cc430cb88f6d10d9684f00 (DEFAULT_TABLE table=default.Table_01d2ed1a594948c4995e0b934e4d2860 [id=e9f106d1b737444baaea6e97a8a3884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.035521 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 111) took 1191 ms. Trace:
I20250110 20:57:10.038076 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 48e53d95ef8645c0bd123138655305ac (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.039767 5846 raft_consensus.cc:491] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.040091 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 371fb7408f0149c8b00659e09ebaf48a (DEFAULT_TABLE table=default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.040212 5846 raft_consensus.cc:513] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.038012 5773 rpcz_store.cc:276] 0110 20:57:08.844382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.845471 (+ 1089us) service_pool.cc:225] Handling call
0110 20:57:09.712085 (+866614us) ts_tablet_manager.cc:1145] Deleting tablet 70ac7d62f978433dad004f2908c4bbc4
0110 20:57:09.712126 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:09.822987 (+110861us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:09.894854 (+ 71867us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.035459 (+140605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":866161}
I20250110 20:57:10.041713 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e508061e93cc430cb88f6d10d9684f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.041977 5846 leader_election.cc:290] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:10.042225 5773 tablet_replica.cc:331] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:10.042971 5773 raft_consensus.cc:2238] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.043464 5773 raft_consensus.cc:2267] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.045559 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 57ef3773a87e4a23a19ac1e5f603d2ce (DEFAULT_TABLE table=default.Table_61b3b8a681344e25acecd34b7455bb08 [id=3392c523ded14b0d899f563eaf1ed148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.050010 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e53d95ef8645c0bd123138655305ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.050585 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ef3773a87e4a23a19ac1e5f603d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.054015 5155 tablet_service.cc:1467] Processing CreateTablet for tablet c44ca47c747544b6afec83b399dcdcb5 (DEFAULT_TABLE table=default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.054332 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371fb7408f0149c8b00659e09ebaf48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.054970 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c44ca47c747544b6afec83b399dcdcb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.056218 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 70ac7d62f978433dad004f2908c4bbc4 (table default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]) successfully deleted
I20250110 20:57:10.064025 5773 ts_tablet_manager.cc:1905] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:10.070829 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 432078ac2fe347e1a55834ca0f77531a (DEFAULT_TABLE table=default.Table_5f0518ca447a401eb8fe8655ecdc9a55 [id=ea236ef3b89c437a84ee6ec4a1cecf15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.071835 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432078ac2fe347e1a55834ca0f77531a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.087194 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 961fef96e9124aafb1c51e287f603b64 with delete_type TABLET_DATA_DELETED (replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:10.087553 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1b57de9e4b4b2ab6c3bce8fe78166a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.089097 5437 tablet_service.cc:1467] Processing CreateTablet for tablet 57ef3773a87e4a23a19ac1e5f603d2ce (DEFAULT_TABLE table=default.Table_61b3b8a681344e25acecd34b7455bb08 [id=3392c523ded14b0d899f563eaf1ed148]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.090066 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961fef96e9124aafb1c51e287f603b64" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.090742 5459 raft_consensus.cc:2463] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:10.091719 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 371fb7408f0149c8b00659e09ebaf48a (DEFAULT_TABLE table=default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.092293 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57ef3773a87e4a23a19ac1e5f603d2ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.093446 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 2644e9180a9548c08cf9e7c4bebb8189 (DEFAULT_TABLE table=default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.094535 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2644e9180a9548c08cf9e7c4bebb8189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.098336 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 48e53d95ef8645c0bd123138655305ac (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.099428 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e53d95ef8645c0bd123138655305ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.102303 5444 tablet_service.cc:1467] Processing CreateTablet for tablet e508061e93cc430cb88f6d10d9684f00 (DEFAULT_TABLE table=default.Table_01d2ed1a594948c4995e0b934e4d2860 [id=e9f106d1b737444baaea6e97a8a3884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.103618 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e508061e93cc430cb88f6d10d9684f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.104841 5833 raft_consensus.cc:491] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.105261 5833 raft_consensus.cc:513] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.107288 5110 leader_election.cc:304] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:10.108283 5779 raft_consensus.cc:491] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.108656 5779 raft_consensus.cc:513] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.109081 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 371fb7408f0149c8b00659e09ebaf48a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.110193 5827 raft_consensus.cc:2798] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:10.110504 5827 raft_consensus.cc:491] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:10.110801 5827 raft_consensus.cc:3054] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.112313 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.112941 5779 leader_election.cc:290] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:10.117926 5110 leader_election.cc:343] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.118417 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9776df98cb54be38a521852634490e7" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.090812 5438 tablet_service.cc:1467] Processing CreateTablet for tablet 432078ac2fe347e1a55834ca0f77531a (DEFAULT_TABLE table=default.Table_5f0518ca447a401eb8fe8655ecdc9a55 [id=ea236ef3b89c437a84ee6ec4a1cecf15]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:10.120509 5257 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 79624.6us
I20250110 20:57:10.121241 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 432078ac2fe347e1a55834ca0f77531a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:10.119778 5110 leader_election.cc:343] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.152820 5594 tablet_bootstrap.cc:492] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:10.107407 5833 leader_election.cc:290] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.153370 5594 ts_tablet_manager.cc:1397] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.280s user 0.012s sys 0.004s
I20250110 20:57:10.157843 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 114) took 1277 ms. Trace:
I20250110 20:57:10.157619 5594 raft_consensus.cc:357] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.158226 5595 rpcz_store.cc:276] 0110 20:57:08.880754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.880886 (+ 132us) service_pool.cc:225] Handling call
0110 20:57:09.788893 (+908007us) ts_tablet_manager.cc:1145] Deleting tablet 78538b2e7bcc45c7afaa145bad6e5813
0110 20:57:09.788930 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:09.888237 (+ 99307us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:09.991728 (+103491us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.157778 (+166050us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":907480}
I20250110 20:57:10.159054 5594 raft_consensus.cc:383] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.159649 5595 tablet_replica.cc:331] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:10.159703 5594 raft_consensus.cc:738] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.160938 5594 consensus_queue.cc:260] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.162127 5597 tablet_bootstrap.cc:492] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:10.162465 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:10.163034 5323 raft_consensus.cc:2463] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
W20250110 20:57:10.162319 5255 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.165673 5597 ts_tablet_manager.cc:1397] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.272s user 0.010s sys 0.004s
I20250110 20:57:10.167289 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 78538b2e7bcc45c7afaa145bad6e5813 (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]) successfully deleted
I20250110 20:57:10.168548 5595 raft_consensus.cc:2238] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.168247 5597 raft_consensus.cc:357] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.168850 5597 raft_consensus.cc:383] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.169070 5595 raft_consensus.cc:2267] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.170876 5595 ts_tablet_manager.cc:1905] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:10.171334 5594 ts_tablet_manager.cc:1428] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.018s user 0.004s sys 0.002s
I20250110 20:57:10.176389 5846 raft_consensus.cc:491] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.176995 5594 tablet_bootstrap.cc:492] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:10.176836 5846 raft_consensus.cc:513] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:10.178930 5846 leader_election.cc:290] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:10.181639 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf0c87159574bfcbc7a34abe508f9d6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:10.182886 5255 leader_election.cc:343] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.199609 5592 ts_tablet_manager.cc:1918] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:10.200001 5592 log.cc:1198] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/477b09061a4a4f278720168872a83258
I20250110 20:57:10.200749 5592 ts_tablet_manager.cc:1939] T 477b09061a4a4f278720168872a83258 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:10.172919 5833 consensus_queue.cc:1035] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:10.179697 5111 leader_election.cc:304] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
W20250110 20:57:10.210330 5111 leader_election.cc:343] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: d9776df98cb54be38a521852634490e7
I20250110 20:57:10.210702 5111 leader_election.cc:304] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:10.184247 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1b57de9e4b4b2ab6c3bce8fe78166a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:10.212111 5779 raft_consensus.cc:2798] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250110 20:57:10.212303 5257 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.212416 5779 raft_consensus.cc:491] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:10.212700 5779 raft_consensus.cc:3054] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.212651 5257 leader_election.cc:304] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:10.213512 5600 tablet_bootstrap.cc:492] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:10.213912 5600 ts_tablet_manager.cc:1397] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.455s user 0.111s sys 0.126s
I20250110 20:57:10.216557 5594 tablet_bootstrap.cc:654] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.184711 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf0c87159574bfcbc7a34abe508f9d6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:10.218269 5257 leader_election.cc:343] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.219024 5257 leader_election.cc:304] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:10.220786 5833 raft_consensus.cc:2743] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.198540 5827 raft_consensus.cc:513] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.223326 5827 leader_election.cc:290] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.224725 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "961fef96e9124aafb1c51e287f603b64" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:10.225250 5459 raft_consensus.cc:3054] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.226675 5597 raft_consensus.cc:738] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.229815 5597 consensus_queue.cc:260] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.230618 5846 raft_consensus.cc:491] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.230858 5600 raft_consensus.cc:357] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.231487 5600 raft_consensus.cc:383] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.231108 5846 raft_consensus.cc:513] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.231750 5600 raft_consensus.cc:738] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.232270 5600 consensus_queue.cc:260] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.238255 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde470279464496aa7efdee3800754f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.213537 5874 raft_consensus.cc:2743] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.240049 5597 ts_tablet_manager.cc:1428] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.074s user 0.004s sys 0.000s
W20250110 20:57:10.239965 5255 leader_election.cc:343] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.240351 5874 raft_consensus.cc:2743] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.240604 5846 leader_election.cc:290] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:10.241010 5846 raft_consensus.cc:491] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.245007 5846 raft_consensus.cc:513] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.246945 5846 leader_election.cc:290] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:10.247871 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6914e258314423c859eb31e85853274" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.248468 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde470279464496aa7efdee3800754f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:10.249136 5255 leader_election.cc:343] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.251394 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6914e258314423c859eb31e85853274" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:10.252739 5257 leader_election.cc:343] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.252847 5597 tablet_bootstrap.cc:492] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:10.253144 5257 leader_election.cc:304] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:10.254069 5257 leader_election.cc:343] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.254426 5257 leader_election.cc:304] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:10.255234 5880 raft_consensus.cc:2743] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.256033 5874 raft_consensus.cc:2743] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.255699 5880 raft_consensus.cc:491] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.260300 5880 raft_consensus.cc:513] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:10.260655 5111 leader_election.cc:343] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 961fef96e9124aafb1c51e287f603b64
I20250110 20:57:10.261322 5876 consensus_queue.cc:1035] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:10.262986 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a666892d04e97a4b9967e38ab0946" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.263885 5880 leader_election.cc:290] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:10.264879 5600 ts_tablet_manager.cc:1428] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.051s user 0.005s sys 0.002s
W20250110 20:57:10.265971 5255 leader_election.cc:343] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.264106 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a666892d04e97a4b9967e38ab0946" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:10.269376 4894 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 2c1ca1670916401ab0a43685df1b6759
I20250110 20:57:10.270696 4894 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 159) took 1148 ms. Trace:
I20250110 20:57:10.270959 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet c6914e258314423c859eb31e85853274
I20250110 20:57:10.271462 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1ca1670916401ab0a43685df1b6759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:10.272215 4900 catalog_manager.cc:6596] ReplaceTablet: tablet c6914e258314423c859eb31e85853274 deleted and replaced by tablet a687769f69d844cc8564e61dc324d2b2
I20250110 20:57:10.273720 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet c6914e258314423c859eb31e85853274 with delete_type TABLET_DATA_DELETED (replaced by tablet a687769f69d844cc8564e61dc324d2b2) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:10.274295 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1ca1670916401ab0a43685df1b6759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:10.278522 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet c6914e258314423c859eb31e85853274 with delete_type TABLET_DATA_DELETED (replaced by tablet a687769f69d844cc8564e61dc324d2b2) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:10.279237 5257 leader_election.cc:343] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.279708 5257 leader_election.cc:304] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:10.277261 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 2d634add82994f74960cd2b8373a4dc6
I20250110 20:57:10.280335 5880 raft_consensus.cc:2743] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.281081 4896 catalog_manager.cc:6596] ReplaceTablet: tablet 2d634add82994f74960cd2b8373a4dc6 deleted and replaced by tablet 33db8a2fdd9c4fde985345604fd04215
I20250110 20:57:10.281353 4896 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36568 (request call id 160) took 1036 ms. Trace:
I20250110 20:57:10.270996 4894 rpcz_store.cc:276] 0110 20:57:09.121894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.121987 (+ 93us) service_pool.cc:225] Handling call
0110 20:57:09.123208 (+ 1221us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:09.196879 (+ 73671us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:09.197001 (+ 122us) catalog_manager.cc:2762] Locking tablets
0110 20:57:09.974782 (+777781us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:10.268977 (+294195us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:10.269189 (+ 212us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:10.269293 (+ 104us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:10.270645 (+ 1352us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:10.012309 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:10.012629 (+ 320us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:10.012651 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:10.012668 (+ 17us) tablet.cc:659] Decoding operations
0110 20:57:10.012946 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:10.012983 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:10.013004 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:10.013261 (+ 257us) tablet.cc:698] Row locks acquired
0110 20:57:10.013274 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:10.013375 (+ 101us) write_op.cc:270] Start()
0110 20:57:10.013477 (+ 102us) write_op.cc:276] Timestamp: P: 1736542630013283 usec, L: 0
0110 20:57:10.013494 (+ 17us) op_driver.cc:348] REPLICATION: starting
0110 20:57:10.013956 (+ 462us) log.cc:844] Serialized 731 byte log entry
0110 20:57:10.267228 (+253272us) op_driver.cc:464] REPLICATION: finished
0110 20:57:10.267491 (+ 263us) write_op.cc:301] APPLY: starting
0110 20:57:10.267561 (+ 70us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:10.267673 (+ 112us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:10.267786 (+ 113us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:10.268013 (+ 227us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:10.268165 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:10.268209 (+ 44us) write_op.cc:312] APPLY: finished
0110 20:57:10.268388 (+ 179us) log.cc:844] Serialized 29 byte log entry
0110 20:57:10.268540 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:10.268607 (+ 67us) write_op.cc:454] Released schema lock
0110 20:57:10.268637 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:10.268833 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:10.268855 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":1323,"apply.run_wall_time_us":1324,"num_ops":2,"prepare.queue_time_us":35924,"prepare.run_cpu_time_us":1884,"prepare.run_wall_time_us":1885,"raft.queue_time_us":159822,"raft.run_cpu_time_us":893,"raft.run_wall_time_us":893,"replication_time_us":253625}]]}
I20250110 20:57:10.285005 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet c6914e258314423c859eb31e85853274 with delete_type TABLET_DATA_DELETED (replaced by tablet a687769f69d844cc8564e61dc324d2b2) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:10.285786 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1ca1670916401ab0a43685df1b6759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:10.287315 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d634add82994f74960cd2b8373a4dc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 33db8a2fdd9c4fde985345604fd04215) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:10.281627 4896 rpcz_store.cc:276] 0110 20:57:09.245019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.245150 (+ 131us) service_pool.cc:225] Handling call
0110 20:57:10.281308 (+1036158us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:10.015986 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:10.016283 (+ 297us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:10.016303 (+ 20us) write_op.cc:435] Acquired schema lock
0110 20:57:10.016316 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:10.016549 (+ 233us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:10.016579 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:10.016600 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:10.016755 (+ 155us) tablet.cc:698] Row locks acquired
0110 20:57:10.016779 (+ 24us) write_op.cc:260] PREPARE: finished
0110 20:57:10.016862 (+ 83us) write_op.cc:270] Start()
0110 20:57:10.016925 (+ 63us) write_op.cc:276] Timestamp: P: 1736542630016773 usec, L: 0
0110 20:57:10.016939 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:10.017293 (+ 354us) log.cc:844] Serialized 628 byte log entry
0110 20:57:10.267672 (+250379us) op_driver.cc:464] REPLICATION: finished
0110 20:57:10.274196 (+ 6524us) write_op.cc:301] APPLY: starting
0110 20:57:10.274245 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:10.274341 (+ 96us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:10.274432 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:10.274704 (+ 272us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:10.276321 (+ 1617us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:10.276365 (+ 44us) write_op.cc:312] APPLY: finished
0110 20:57:10.276527 (+ 162us) log.cc:844] Serialized 29 byte log entry
0110 20:57:10.276667 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:10.276723 (+ 56us) write_op.cc:454] Released schema lock
0110 20:57:10.276755 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:10.276952 (+ 197us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:10.276976 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6398,"apply.run_cpu_time_us":1313,"apply.run_wall_time_us":2725,"num_ops":2,"prepare.queue_time_us":36279,"prepare.run_cpu_time_us":1557,"prepare.run_wall_time_us":160388,"raft.queue_time_us":156845,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"replication_time_us":250665,"spinlock_wait_cycles":119552}]]}
I20250110 20:57:10.291146 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
I20250110 20:57:10.291330 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_0b1d1f4267df44e6a2555f1080545d59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:10.291709 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
W20250110 20:57:10.292681 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b1d1f4267df44e6a2555f1080545d59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:10.293684 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet cd0a666892d04e97a4b9967e38ab0946
I20250110 20:57:10.295850 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_70b6412a11364f1389adbac0db42cdb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:10.296483 5595 ts_tablet_manager.cc:1918] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:10.296831 5595 log.cc:1198] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f1fafed70adb496d97acfd5fea356292
W20250110 20:57:10.297108 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70b6412a11364f1389adbac0db42cdb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:10.297397 5595 ts_tablet_manager.cc:1939] T f1fafed70adb496d97acfd5fea356292 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:10.298156 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet cd0a666892d04e97a4b9967e38ab0946 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:10.298877 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet cd0a666892d04e97a4b9967e38ab0946 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:10.298933 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 101) took 2307 ms. Trace:
I20250110 20:57:10.299300 5592 rpcz_store.cc:276] 0110 20:57:07.991850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.050107 (+ 58257us) service_pool.cc:225] Handling call
0110 20:57:09.736403 (+1686296us) ts_tablet_manager.cc:1145] Deleting tablet 477b09061a4a4f278720168872a83258
0110 20:57:09.736439 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:09.745349 (+ 8910us) spinlock_profiling.cc:243] Waited 6.56 ms on lock 0x7b5400100254. stack: 00007f2677942853 00007f2677942750 00007f26774b7584 00000000004fcd24 00000000004fccb9 00007f267e8426d1 00007f267e8c5352 00007f267ece21ac 00007f267f8f2d5a 00007f267f922ca0 00007f267f909e6d 00007f267f909e19 00007f267f909da9 00007f267f909d71 00007f267f908f9d 000000000059ea04
0110 20:57:10.128870 (+383521us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.199299 (+ 70429us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.298869 (+ 99570us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15084416,"tablet-delete.queue_time_us":1672965}
I20250110 20:57:10.300645 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 477b09061a4a4f278720168872a83258 (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]) successfully deleted
I20250110 20:57:10.300704 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 113) took 1677 ms. Trace:
I20250110 20:57:10.300969 5592 rpcz_store.cc:276] 0110 20:57:08.623066 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.623189 (+ 123us) service_pool.cc:225] Handling call
0110 20:57:10.300341 (+1677152us) ts_tablet_manager.cc:1145] Deleting tablet 1ac07ac6fc9f477ba7f9accc5533be39
0110 20:57:10.300405 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.300647 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1676632}
I20250110 20:57:10.301323 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a1f70418cc2f433d90434c4621fd49d0" } new_table_name: "default.Table_a73e1fb622794aa4a8a231555b3d53ca" modify_external_catalogs: true
I20250110 20:57:10.301450 5880 raft_consensus.cc:491] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.302078 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 114) took 1487 ms. Trace:
I20250110 20:57:10.301837 5880 raft_consensus.cc:513] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.302337 5592 rpcz_store.cc:276] 0110 20:57:08.815183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.815316 (+ 133us) service_pool.cc:225] Handling call
0110 20:57:10.301716 (+1486400us) ts_tablet_manager.cc:1145] Deleting tablet 8d67e94c8f9a414499b5e20a5cc0ead3
0110 20:57:10.301759 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.302034 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1485892}
I20250110 20:57:10.303313 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 115) took 1417 ms. Trace:
I20250110 20:57:10.303539 5880 leader_election.cc:290] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:10.303571 5592 rpcz_store.cc:276] 0110 20:57:08.885559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.885684 (+ 125us) service_pool.cc:225] Handling call
0110 20:57:10.303007 (+1417323us) ts_tablet_manager.cc:1145] Deleting tablet 78538b2e7bcc45c7afaa145bad6e5813
0110 20:57:10.303061 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.303271 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360963}
I20250110 20:57:10.304586 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 116) took 1393 ms. Trace:
I20250110 20:57:10.304854 5592 rpcz_store.cc:276] 0110 20:57:08.911337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.911466 (+ 129us) service_pool.cc:225] Handling call
0110 20:57:10.304272 (+1392806us) ts_tablet_manager.cc:1145] Deleting tablet f1fafed70adb496d97acfd5fea356292
0110 20:57:10.304313 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.304542 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1357840}
I20250110 20:57:10.305243 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cbb46ea11c4e7d8bc5993ccdc52247" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:10.306231 4891 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 41de354807004e26bfb147f1d420037e consensus state has no leader
W20250110 20:57:10.306548 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 1ac07ac6fc9f477ba7f9accc5533be39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ac07ac6fc9f477ba7f9accc5533be39 already in progress: creating tablet
W20250110 20:57:10.306507 5255 leader_election.cc:343] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:10.307103 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d67e94c8f9a414499b5e20a5cc0ead3 already in progress: creating tablet
I20250110 20:57:10.307437 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:10.307632 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 78538b2e7bcc45c7afaa145bad6e5813 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78538b2e7bcc45c7afaa145bad6e5813 already in progress: creating tablet
I20250110 20:57:10.299441 5459 raft_consensus.cc:2463] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:10.311883 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d634add82994f74960cd2b8373a4dc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 33db8a2fdd9c4fde985345604fd04215) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:10.314226 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet f1fafed70adb496d97acfd5fea356292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1fafed70adb496d97acfd5fea356292 already in progress: creating tablet
I20250110 20:57:10.314698 5592 tablet_replica.cc:331] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:10.314793 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 637a39e4264b40dd9480dc13e37359d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 637a39e4264b40dd9480dc13e37359d9 already in progress: creating tablet
I20250110 20:57:10.315431 5592 raft_consensus.cc:2238] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.315899 5592 raft_consensus.cc:2267] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.317555 5592 ts_tablet_manager.cc:1905] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:10.323092 5594 tablet_bootstrap.cc:492] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:10.323549 5594 ts_tablet_manager.cc:1397] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.147s user 0.012s sys 0.013s
W20250110 20:57:10.325196 4892 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 36) took 1097 ms (client timeout 1000 ms). Trace:
I20250110 20:57:10.325721 5594 raft_consensus.cc:357] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.326339 5594 raft_consensus.cc:383] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.326588 5594 raft_consensus.cc:738] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.327116 5594 consensus_queue.cc:260] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.330864 5600 tablet_bootstrap.cc:492] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:10.329967 4892 rpcz_store.cc:269] 0110 20:57:09.227494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.227620 (+ 126us) service_pool.cc:225] Handling call
0110 20:57:10.325138 (+1097518us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:10.269735 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:10.272242 (+ 2507us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:10.272264 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:10.272278 (+ 14us) tablet.cc:659] Decoding operations
0110 20:57:10.272509 (+ 231us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:10.272540 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:10.272561 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:10.272683 (+ 122us) tablet.cc:698] Row locks acquired
0110 20:57:10.272695 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:10.272780 (+ 85us) write_op.cc:270] Start()
0110 20:57:10.272864 (+ 84us) write_op.cc:276] Timestamp: P: 1736542630272715 usec, L: 0
0110 20:57:10.272878 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:10.305481 (+ 32603us) log.cc:844] Serialized 533 byte log entry
0110 20:57:10.307275 (+ 1794us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd4796d1 00007f3bfd4f0198 00007f3bfd95c3b1 00007f3bfd95bb88 00007f3bfd960821 00007f3bfd9607d9 00007f3bfd960769 00007f3bfd960731 00007f3bfd95f95d 000000000059ea04 000000000059e839
0110 20:57:10.315097 (+ 7822us) op_driver.cc:464] REPLICATION: finished
0110 20:57:10.315561 (+ 464us) write_op.cc:301] APPLY: starting
0110 20:57:10.315605 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:10.315686 (+ 81us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:10.315704 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:10.315904 (+ 200us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:10.316016 (+ 112us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:10.316043 (+ 27us) write_op.cc:312] APPLY: finished
0110 20:57:10.316173 (+ 130us) log.cc:844] Serialized 23 byte log entry
0110 20:57:10.316294 (+ 121us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:10.316350 (+ 56us) write_op.cc:454] Released schema lock
0110 20:57:10.316381 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:10.327948 (+ 11567us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:10.327977 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":337,"apply.run_cpu_time_us":937,"apply.run_wall_time_us":12354,"num_ops":1,"prepare.queue_time_us":261451,"prepare.run_cpu_time_us":2598,"prepare.run_wall_time_us":37701,"raft.queue_time_us":90,"raft.run_cpu_time_us":540,"raft.run_wall_time_us":539,"replication_time_us":42136,"spinlock_wait_cycles":73812352}]]}
I20250110 20:57:10.333365 5594 ts_tablet_manager.cc:1428] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20250110 20:57:10.339319 5594 tablet_bootstrap.cc:492] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:10.336915 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet f1fafed70adb496d97acfd5fea356292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:10.351320 5779 raft_consensus.cc:513] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.354151 5600 tablet_bootstrap.cc:654] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.354406 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:10.355103 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 477b09061a4a4f278720168872a83258 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:10.355553 5779 leader_election.cc:290] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.355685 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d634add82994f74960cd2b8373a4dc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 33db8a2fdd9c4fde985345604fd04215) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:10.356263 5773 ts_tablet_manager.cc:1918] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:10.356405 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet a7e7ffb063594128ba137f9aea86bf15 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:10.356619 5773 log.cc:1198] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/9155a626d26541ce9d0cf31508f19f15
I20250110 20:57:10.357221 5597 tablet_bootstrap.cc:654] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.357497 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet cd0a666892d04e97a4b9967e38ab0946 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:10.357740 5773 ts_tablet_manager.cc:1939] T 9155a626d26541ce9d0cf31508f19f15 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:10.353847 5594 tablet_bootstrap.cc:654] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.367017 5110 leader_election.cc:304] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:10.367473 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:10.368345 5779 raft_consensus.cc:2798] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:10.369149 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:10.370126 5323 raft_consensus.cc:3054] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.372473 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cbb46ea11c4e7d8bc5993ccdc52247" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:10.368752 5779 raft_consensus.cc:695] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:10.373754 5779 consensus_queue.cc:237] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.377918 5594 tablet_bootstrap.cc:492] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:10.378301 5594 ts_tablet_manager.cc:1397] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.003s
I20250110 20:57:10.380719 5594 raft_consensus.cc:357] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.381325 5594 raft_consensus.cc:383] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.381577 5594 raft_consensus.cc:738] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.382223 5594 consensus_queue.cc:260] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.386690 5594 ts_tablet_manager.cc:1428] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250110 20:57:10.387580 5594 tablet_bootstrap.cc:492] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
W20250110 20:57:10.392604 5110 leader_election.cc:343] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.394110 5827 raft_consensus.cc:491] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.394536 5827 raft_consensus.cc:513] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.396804 5827 leader_election.cc:290] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.397768 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 115) took 1483 ms. Trace:
W20250110 20:57:10.398140 5257 leader_election.cc:343] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.398095 5595 rpcz_store.cc:276] 0110 20:57:08.914272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.914401 (+ 129us) service_pool.cc:225] Handling call
0110 20:57:10.159499 (+1245098us) ts_tablet_manager.cc:1145] Deleting tablet f1fafed70adb496d97acfd5fea356292
0110 20:57:10.159548 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.224763 (+ 65215us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.296281 (+ 71518us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.397704 (+101423us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8149,"tablet-delete.queue_time_us":1244543}
I20250110 20:57:10.398648 5257 leader_election.cc:304] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:10.399174 5595 tablet_replica.cc:331] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:10.399441 5880 raft_consensus.cc:2743] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.399844 5595 raft_consensus.cc:2238] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.402242 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet f1fafed70adb496d97acfd5fea356292 (table default.Table_5107a94b926b4518baaf9bacd04d2f46 [id=e2dbf27ca6424662a7a14c9a9ae5587d]) successfully deleted
I20250110 20:57:10.402457 5595 raft_consensus.cc:2267] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.404003 5595 ts_tablet_manager.cc:1905] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:10.407542 5111 leader_election.cc:343] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:10.409634 5323 raft_consensus.cc:2463] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:10.411504 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 112) took 1566 ms. Trace:
I20250110 20:57:10.411818 5773 rpcz_store.cc:276] 0110 20:57:08.844620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.845997 (+ 1377us) service_pool.cc:225] Handling call
0110 20:57:10.042073 (+1196076us) ts_tablet_manager.cc:1145] Deleting tablet 9155a626d26541ce9d0cf31508f19f15
0110 20:57:10.042125 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.201367 (+159242us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.356033 (+154666us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.411451 (+ 55418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195625}
I20250110 20:57:10.413007 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 110) took 1568 ms. Trace:
I20250110 20:57:10.414505 5600 tablet_bootstrap.cc:492] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:10.414438 5111 leader_election.cc:304] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:10.415973 5600 ts_tablet_manager.cc:1397] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.000s
I20250110 20:57:10.418989 5827 raft_consensus.cc:2798] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:10.421449 5597 tablet_bootstrap.cc:492] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:10.421869 5597 ts_tablet_manager.cc:1397] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.181s user 0.008s sys 0.002s
I20250110 20:57:10.423332 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 9155a626d26541ce9d0cf31508f19f15 (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]) successfully deleted
I20250110 20:57:10.424012 5597 raft_consensus.cc:357] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.424662 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a520230d9fd4a74914e46f784dcf747" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.432904 5594 tablet_bootstrap.cc:654] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.438863 5600 raft_consensus.cc:357] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.441520 5600 raft_consensus.cc:383] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.441835 5600 raft_consensus.cc:738] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:10.427366 5110 leader_election.cc:343] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.444407 5110 leader_election.cc:304] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:10.413314 5773 rpcz_store.cc:276] 0110 20:57:08.844142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.845374 (+ 1232us) service_pool.cc:225] Handling call
0110 20:57:10.412776 (+1567402us) ts_tablet_manager.cc:1145] Deleting tablet 5793d909c6364047abc85ea5f993c521
0110 20:57:10.412813 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.412961 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565694}
I20250110 20:57:10.446069 5773 tablet_replica.cc:331] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:10.446882 5773 raft_consensus.cc:2238] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.443476 5600 consensus_queue.cc:260] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.447363 5773 raft_consensus.cc:2267] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.449345 5597 raft_consensus.cc:383] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.449658 5597 raft_consensus.cc:738] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.450052 5600 ts_tablet_manager.cc:1428] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
W20250110 20:57:10.450430 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 5793d909c6364047abc85ea5f993c521 because the tablet was not found. No further retry: Not found: Tablet not found: 5793d909c6364047abc85ea5f993c521
I20250110 20:57:10.451370 5597 consensus_queue.cc:260] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.431525 5779 consensus_queue.cc:1035] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:10.456612 5773 ts_tablet_manager.cc:1905] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:10.459609 5833 raft_consensus.cc:2743] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.460211 5833 consensus_queue.cc:1035] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:10.462034 5600 tablet_bootstrap.cc:492] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:10.471184 5594 tablet_bootstrap.cc:492] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:10.471546 5594 ts_tablet_manager.cc:1397] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.000s
I20250110 20:57:10.473611 5594 raft_consensus.cc:357] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.474210 5594 raft_consensus.cc:383] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.474426 5594 raft_consensus.cc:738] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.478376 5827 raft_consensus.cc:695] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:10.481663 5594 consensus_queue.cc:260] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.486408 5594 ts_tablet_manager.cc:1428] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.015s user 0.006s sys 0.001s
I20250110 20:57:10.488934 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_8d1a07d9e4cf46ec81956254772651a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:10.487360 5600 tablet_bootstrap.cc:654] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:10.490353 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d1a07d9e4cf46ec81956254772651a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:10.492417 5594 tablet_bootstrap.cc:492] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:10.494941 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8e63a6ded00f470d94804f6f15343b29" } modify_external_catalogs: true
I20250110 20:57:10.495405 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8e63a6ded00f470d94804f6f15343b29" } modify_external_catalogs: true
I20250110 20:57:10.498286 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e17e3d34a10d4c05b4ae644cd35c9263" } modify_external_catalogs: true
I20250110 20:57:10.498728 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e17e3d34a10d4c05b4ae644cd35c9263" } modify_external_catalogs: true
I20250110 20:57:10.480902 5827 consensus_queue.cc:237] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.501860 5592 ts_tablet_manager.cc:1918] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:10.502190 5592 log.cc:1198] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/5793d909c6364047abc85ea5f993c521
I20250110 20:57:10.502748 5592 ts_tablet_manager.cc:1939] T 5793d909c6364047abc85ea5f993c521 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
W20250110 20:57:10.503712 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a7a09ea54e79485b9892d52d0e04da85 consensus state has no leader
I20250110 20:57:10.507393 5597 ts_tablet_manager.cc:1428] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.085s user 0.004s sys 0.000s
I20250110 20:57:10.508199 5597 tablet_bootstrap.cc:492] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:10.509234 5594 tablet_bootstrap.cc:654] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.519994 5600 tablet_bootstrap.cc:492] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:10.520385 5600 ts_tablet_manager.cc:1397] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.000s
I20250110 20:57:10.525509 5600 raft_consensus.cc:357] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.526162 5600 raft_consensus.cc:383] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.526408 5600 raft_consensus.cc:738] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.526994 5600 consensus_queue.cc:260] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.539393 5600 ts_tablet_manager.cc:1428] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.019s user 0.005s sys 0.002s
I20250110 20:57:10.542199 5600 tablet_bootstrap.cc:492] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:10.547940 5597 tablet_bootstrap.cc:654] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.553622 5600 tablet_bootstrap.cc:654] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.582906 5600 tablet_bootstrap.cc:492] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:10.583298 5600 ts_tablet_manager.cc:1397] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.003s
I20250110 20:57:10.585486 5600 raft_consensus.cc:357] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.586148 5600 raft_consensus.cc:383] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.586388 5600 raft_consensus.cc:738] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.587090 5600 consensus_queue.cc:260] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.589385 5600 ts_tablet_manager.cc:1428] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250110 20:57:10.591022 5600 tablet_bootstrap.cc:492] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:10.591305 4900 master_service.cc:946] ReplaceTablet: received request to replace tablet 527bbddda8184a6eb67c2411ae50db63 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:10.598019 5595 ts_tablet_manager.cc:1918] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:10.598343 5595 log.cc:1198] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/2967c06715e241dcbbe784eed2373704
I20250110 20:57:10.598618 5597 tablet_bootstrap.cc:492] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:10.598903 5595 ts_tablet_manager.cc:1939] T 2967c06715e241dcbbe784eed2373704 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:10.599093 5597 ts_tablet_manager.cc:1397] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.091s user 0.013s sys 0.005s
I20250110 20:57:10.600697 4895 master_service.cc:946] ReplaceTablet: received request to replace tablet fcc7b1acf4114bf69d801437855760df from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:10.601145 5597 raft_consensus.cc:357] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.601851 5597 raft_consensus.cc:383] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.602092 5597 raft_consensus.cc:738] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.602612 5597 consensus_queue.cc:260] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.606582 5600 tablet_bootstrap.cc:654] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.607753 5597 ts_tablet_manager.cc:1428] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250110 20:57:10.613938 5597 tablet_bootstrap.cc:492] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:10.624171 5600 tablet_bootstrap.cc:492] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:10.624547 5600 ts_tablet_manager.cc:1397] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.005s
I20250110 20:57:10.626691 5600 raft_consensus.cc:357] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.627332 5600 raft_consensus.cc:383] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.627590 5600 raft_consensus.cc:738] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:10.628068 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.252:37233, user_credentials={real_user=slave}} blocked reactor thread for 74366.2us
I20250110 20:57:10.629135 5597 tablet_bootstrap.cc:654] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.638979 5600 consensus_queue.cc:260] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.644249 5600 ts_tablet_manager.cc:1428] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.019s user 0.006s sys 0.001s
I20250110 20:57:10.647384 5600 tablet_bootstrap.cc:492] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:10.650981 5827 raft_consensus.cc:491] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.651429 5827 raft_consensus.cc:513] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.653157 5827 leader_election.cc:290] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.653738 5852 raft_consensus.cc:491] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.654150 5852 raft_consensus.cc:513] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.655555 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6eca349514b9fb0fff7cd6230027c" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.655974 5852 leader_election.cc:290] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.656144 5459 raft_consensus.cc:2463] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:10.657256 5110 leader_election.cc:304] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:10.659695 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6eca349514b9fb0fff7cd6230027c" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:10.660264 5323 raft_consensus.cc:2463] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:10.656400 5852 raft_consensus.cc:491] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 20:57:10.661744 5398 leader_election.cc:343] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 78538b2e7bcc45c7afaa145bad6e5813
I20250110 20:57:10.662675 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78538b2e7bcc45c7afaa145bad6e5813" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:10.662889 5592 tablet_replica.cc:331] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:10.663533 5592 raft_consensus.cc:2238] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.663913 5592 raft_consensus.cc:2267] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.664161 5827 raft_consensus.cc:2798] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:10.664464 5827 raft_consensus.cc:491] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:10.664746 5827 raft_consensus.cc:3054] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.665627 5852 raft_consensus.cc:513] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.667400 5592 ts_tablet_manager.cc:1905] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:10.663935 5399 leader_election.cc:343] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:10.669886 5399 leader_election.cc:304] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:10.670974 5852 leader_election.cc:290] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:10.672614 5398 leader_election.cc:343] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: d9776df98cb54be38a521852634490e7
W20250110 20:57:05.626168 5646 debug-util.cc:398] Leaking SignalData structure 0x7b0800006bc0 after lost signal to thread 5581
W20250110 20:57:04.098093 5651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:57:10.675364 5646 kernel_stack_watchdog.cc:198] Thread 5581 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 528ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250110 20:57:10.677165 5581 thread.cc:635] OpenStack (cloud detector) Time spent creating pthread: real 6.582s user 0.506s sys 0.805s
W20250110 20:57:10.677547 5581 thread.cc:601] OpenStack (cloud detector) Time spent starting thread: real 6.582s user 0.507s sys 0.805s
W20250110 20:57:10.679749 5656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:10.679893 5600 tablet_bootstrap.cc:654] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.680769 5890 raft_consensus.cc:491] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.681253 5890 raft_consensus.cc:513] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.683173 5890 leader_election.cc:290] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.683604 5581 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250110 20:57:10.684034 5655 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 6587 milliseconds
I20250110 20:57:10.685120 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b65488c884452d876ced1c3d1efb9a" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:10.686544 5111 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 31b65488c884452d876ced1c3d1efb9a
W20250110 20:57:10.687443 5110 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.687783 5110 leader_election.cc:304] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:10.688684 5890 raft_consensus.cc:2743] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.689538 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 5793d909c6364047abc85ea5f993c521 (table default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]) successfully deleted
I20250110 20:57:10.699165 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 119) took 1670 ms. Trace:
I20250110 20:57:10.699489 5595 rpcz_store.cc:276] 0110 20:57:09.029118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.029218 (+ 100us) service_pool.cc:225] Handling call
0110 20:57:10.399046 (+1369828us) ts_tablet_manager.cc:1145] Deleting tablet 2967c06715e241dcbbe784eed2373704
0110 20:57:10.399080 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.498082 (+ 99002us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.597823 (+ 99741us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.699105 (+101282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1368676}
I20250110 20:57:10.700491 5595 tablet_replica.cc:331] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:10.701941 5773 ts_tablet_manager.cc:1918] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:10.702319 5773 log.cc:1198] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/78538b2e7bcc45c7afaa145bad6e5813
I20250110 20:57:10.702970 5773 ts_tablet_manager.cc:1939] T 78538b2e7bcc45c7afaa145bad6e5813 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:10.696951 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9776df98cb54be38a521852634490e7" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:10.704120 5173 raft_consensus.cc:2463] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:10.701211 5595 raft_consensus.cc:2238] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.706410 5581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:10.706482 5595 raft_consensus.cc:2267] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.708226 5595 ts_tablet_manager.cc:1905] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:10.713768 5907 raft_consensus.cc:2743] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:10.716238 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 7309d13f62524b6ab8b4c551728ecb6d (DEFAULT_TABLE table=default.Table_f1a0fe849b9c44a1b4834b9835cb49f9 [id=1c2b9e8ff5d341c380b50023c577d0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.710700 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet fde470279464496aa7efdee3800754f3 (table default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]). Expected version 1 got 0
I20250110 20:57:10.717691 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:10.718093 5155 tablet_service.cc:1467] Processing CreateTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 (DEFAULT_TABLE table=default.Table_e17e3d34a10d4c05b4ae644cd35c9263 [id=99a3fb146486431188b7bc452ea806ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.719751 5399 leader_election.cc:304] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
W20250110 20:57:10.720181 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a687769f69d844cc8564e61dc324d2b2 has no consensus state
I20250110 20:57:10.720479 5907 raft_consensus.cc:2798] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:10.722168 5907 raft_consensus.cc:491] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:10.722488 5907 raft_consensus.cc:3054] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.717274 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7309d13f62524b6ab8b4c551728ecb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.729694 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c813130649cc4df1b483ac4a86b4f0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.730266 5434 tablet_service.cc:1467] Processing CreateTablet for tablet 7309d13f62524b6ab8b4c551728ecb6d (DEFAULT_TABLE table=default.Table_f1a0fe849b9c44a1b4834b9835cb49f9 [id=1c2b9e8ff5d341c380b50023c577d0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.731287 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7309d13f62524b6ab8b4c551728ecb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.734681 5444 tablet_service.cc:1467] Processing CreateTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 (DEFAULT_TABLE table=default.Table_e17e3d34a10d4c05b4ae644cd35c9263 [id=99a3fb146486431188b7bc452ea806ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.735708 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c813130649cc4df1b483ac4a86b4f0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.737507 4899 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 472174c6885b446a839721157914f047 (table default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688]). Expected version 1 got 0
I20250110 20:57:10.738242 4899 catalog_manager.cc:5526] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:10.739419 4899 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]). Expected version 2 got 0
I20250110 20:57:10.740067 4899 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet ae5956b1aa17449497f9695a111ebb66 (table default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]). Expected version 1 got 0
I20250110 20:57:10.740697 4899 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:10.736932 5827 raft_consensus.cc:513] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.743861 5153 tablet_service.cc:1467] Processing CreateTablet for tablet f3104928188c489d93bfd499d148fa5d (DEFAULT_TABLE table=default.Table_338aa5da81e146be96e0f8cc5dcae996 [id=75d13198e6a74915b9955b32e3908f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.744777 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3104928188c489d93bfd499d148fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.745923 5827 leader_election.cc:290] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.749593 5594 tablet_bootstrap.cc:492] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:10.750072 5594 ts_tablet_manager.cc:1397] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.263s user 0.011s sys 0.009s
I20250110 20:57:10.752315 5594 raft_consensus.cc:357] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.753023 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6eca349514b9fb0fff7cd6230027c" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:10.757025 5144 tablet_service.cc:1467] Processing CreateTablet for tablet bae52ea080f54d228b6b76922248ed5a (DEFAULT_TABLE table=default.Table_41125035ac2544fb9a27c3c25133b711 [id=4bf0e84fcbfb4b2a841abc862506a8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.758147 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae52ea080f54d228b6b76922248ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.758599 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 484a3aebaafb4f2aa47f009b773209fe (DEFAULT_TABLE table=default.Table_1498d22e348d4599a5514859d6371910 [id=01e0d2da0c684723bb44ab133d59f983]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.759495 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484a3aebaafb4f2aa47f009b773209fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.761365 5459 raft_consensus.cc:3054] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.767211 5438 tablet_service.cc:1467] Processing CreateTablet for tablet f3104928188c489d93bfd499d148fa5d (DEFAULT_TABLE table=default.Table_338aa5da81e146be96e0f8cc5dcae996 [id=75d13198e6a74915b9955b32e3908f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.768714 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 484a3aebaafb4f2aa47f009b773209fe (DEFAULT_TABLE table=default.Table_1498d22e348d4599a5514859d6371910 [id=01e0d2da0c684723bb44ab133d59f983]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.770154 5443 tablet_service.cc:1467] Processing CreateTablet for tablet bae52ea080f54d228b6b76922248ed5a (DEFAULT_TABLE table=default.Table_41125035ac2544fb9a27c3c25133b711 [id=4bf0e84fcbfb4b2a841abc862506a8c2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:10.770319 5110 consensus_peers.cc:487] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477b09061a4a4f278720168872a83258. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:10.770571 5146 tablet_service.cc:1467] Processing CreateTablet for tablet c57941aee12a4ebea2b47f4727fc6eee (DEFAULT_TABLE table=default.Table_a0178b82e1d8457eb7f165640e222c72 [id=1a67e1de31b3490e8c391f502b5a02bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.771521 5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c57941aee12a4ebea2b47f4727fc6eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.771807 5439 tablet_service.cc:1467] Processing CreateTablet for tablet c57941aee12a4ebea2b47f4727fc6eee (DEFAULT_TABLE table=default.Table_a0178b82e1d8457eb7f165640e222c72 [id=1a67e1de31b3490e8c391f502b5a02bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.780081 5437 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb75e1886e74a719dc09ccad8e1e284 (DEFAULT_TABLE table=default.Table_1c8384b6009d43a89936a5f543415675 [id=76a6826d1a6b46bf8b300259c63eb1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.783310 5581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:10.782704 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb75e1886e74a719dc09ccad8e1e284 (DEFAULT_TABLE table=default.Table_1c8384b6009d43a89936a5f543415675 [id=76a6826d1a6b46bf8b300259c63eb1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.784202 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eb75e1886e74a719dc09ccad8e1e284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.753201 5594 raft_consensus.cc:383] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.785187 5594 raft_consensus.cc:738] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.785863 5594 consensus_queue.cc:260] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.788725 5594 ts_tablet_manager.cc:1428] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20250110 20:57:10.795553 5594 tablet_bootstrap.cc:492] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:10.801594 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 113) took 1914 ms. Trace:
I20250110 20:57:10.801961 5773 rpcz_store.cc:276] 0110 20:57:08.887274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:08.942593 (+ 55319us) service_pool.cc:225] Handling call
0110 20:57:10.445920 (+1503327us) ts_tablet_manager.cc:1145] Deleting tablet 78538b2e7bcc45c7afaa145bad6e5813
0110 20:57:10.445961 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.593383 (+147422us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.701674 (+108291us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.801528 (+ 99854us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420331}
I20250110 20:57:10.803457 5773 tablet_replica.cc:331] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:10.803557 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 78538b2e7bcc45c7afaa145bad6e5813 (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]) successfully deleted
I20250110 20:57:10.804222 5773 raft_consensus.cc:2238] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:10.804651 5773 raft_consensus.cc:2267] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:10.804977 5436 tablet_service.cc:1467] Processing CreateTablet for tablet fb8c87076a8a4b1e86f7e4844e17b1f3 (DEFAULT_TABLE table=default.Table_368911cd6dd24875993ac8c44548a12d [id=e6b770e622ab4e3db1a8927d3d6894be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.805740 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484a3aebaafb4f2aa47f009b773209fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.806268 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3104928188c489d93bfd499d148fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.806771 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae52ea080f54d228b6b76922248ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.807291 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eb75e1886e74a719dc09ccad8e1e284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.807768 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8c87076a8a4b1e86f7e4844e17b1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.808323 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c57941aee12a4ebea2b47f4727fc6eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.821940 5907 raft_consensus.cc:513] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.825289 5907 leader_election.cc:290] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.831722 5594 tablet_bootstrap.cc:654] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.835309 5154 tablet_service.cc:1467] Processing CreateTablet for tablet fb8c87076a8a4b1e86f7e4844e17b1f3 (DEFAULT_TABLE table=default.Table_368911cd6dd24875993ac8c44548a12d [id=e6b770e622ab4e3db1a8927d3d6894be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.835345 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9776df98cb54be38a521852634490e7" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:10.836037 5173 raft_consensus.cc:3054] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:10.836596 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8c87076a8a4b1e86f7e4844e17b1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.837890 5597 tablet_bootstrap.cc:492] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:10.838975 5597 ts_tablet_manager.cc:1397] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.225s user 0.006s sys 0.003s
I20250110 20:57:10.850735 5773 ts_tablet_manager.cc:1905] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:10.856930 5827 raft_consensus.cc:491] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.857333 5827 raft_consensus.cc:513] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.859427 5827 leader_election.cc:290] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.860772 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d634add82994f74960cd2b8373a4dc6" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.860962 5597 raft_consensus.cc:357] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.861579 5597 raft_consensus.cc:383] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:10.861995 5907 raft_consensus.cc:491] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.862375 5907 raft_consensus.cc:513] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.861828 5597 raft_consensus.cc:738] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:10.864015 5907 leader_election.cc:290] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:10.864997 5110 leader_election.cc:343] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:10.867830 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:10.869421 5597 consensus_queue.cc:260] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:10.874608 4887 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet a687769f69d844cc8564e61dc324d2b2 (table default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a687769f69d844cc8564e61dc324d2b2 has no consensus state
I20250110 20:57:10.875012 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ec0889bc4a894c00bcc098b12fa053fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:10.876395 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec0889bc4a894c00bcc098b12fa053fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:10.877818 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_f8573af69b094101bbd0f64ce7c1716c" } new_table_name: "default.Table_2a36a6d3c1c843eba552b6fd97984099" modify_external_catalogs: true
I20250110 20:57:10.885975 5597 ts_tablet_manager.cc:1428] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20250110 20:57:10.887108 5597 tablet_bootstrap.cc:492] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:10.889498 5581 hybrid_clock.cc:648] HybridClock initialized: now 1736542630889403 us; error 101 us; skew 500 ppm
I20250110 20:57:10.890619 5581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:10.901093 5459 raft_consensus.cc:2463] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:10.902400 5110 leader_election.cc:304] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:10.903113 5890 raft_consensus.cc:2798] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:10.903527 5890 raft_consensus.cc:695] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:10.904197 5890 consensus_queue.cc:237] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.911687 5907 raft_consensus.cc:491] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.912087 5907 raft_consensus.cc:513] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:10.913846 5907 leader_election.cc:290] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:10.914705 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 2967c06715e241dcbbe784eed2373704 (table default.Table_6d29a7293ace46b186d68a71d0a5a688 [id=4f67b22c80ae49b79ddc245c7e304307]) successfully deleted
I20250110 20:57:10.915846 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 7309d13f62524b6ab8b4c551728ecb6d (DEFAULT_TABLE table=default.Table_f1a0fe849b9c44a1b4834b9835cb49f9 [id=1c2b9e8ff5d341c380b50023c577d0aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.916867 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7309d13f62524b6ab8b4c551728ecb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.918507 5292 tablet_service.cc:1467] Processing CreateTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 (DEFAULT_TABLE table=default.Table_e17e3d34a10d4c05b4ae644cd35c9263 [id=99a3fb146486431188b7bc452ea806ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.919754 5296 tablet_service.cc:1467] Processing CreateTablet for tablet f3104928188c489d93bfd499d148fa5d (DEFAULT_TABLE table=default.Table_338aa5da81e146be96e0f8cc5dcae996 [id=75d13198e6a74915b9955b32e3908f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.920987 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 484a3aebaafb4f2aa47f009b773209fe (DEFAULT_TABLE table=default.Table_1498d22e348d4599a5514859d6371910 [id=01e0d2da0c684723bb44ab133d59f983]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.921936 5880 raft_consensus.cc:491] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:10.922313 5880 raft_consensus.cc:513] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:10.931581 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3104928188c489d93bfd499d148fa5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:10.946487 5398 leader_election.cc:343] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: f1fafed70adb496d97acfd5fea356292
I20250110 20:57:10.949759 5293 tablet_service.cc:1467] Processing CreateTablet for tablet bae52ea080f54d228b6b76922248ed5a (DEFAULT_TABLE table=default.Table_41125035ac2544fb9a27c3c25133b711 [id=4bf0e84fcbfb4b2a841abc862506a8c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.952558 5880 leader_election.cc:290] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:10.947621 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 484a3aebaafb4f2aa47f009b773209fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:10.977730 5301 tablet_service.cc:1467] Processing CreateTablet for tablet c57941aee12a4ebea2b47f4727fc6eee (DEFAULT_TABLE table=default.Table_a0178b82e1d8457eb7f165640e222c72 [id=1a67e1de31b3490e8c391f502b5a02bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.978057 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 5eb75e1886e74a719dc09ccad8e1e284 (DEFAULT_TABLE table=default.Table_1c8384b6009d43a89936a5f543415675 [id=76a6826d1a6b46bf8b300259c63eb1b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:10.944654 5321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d634add82994f74960cd2b8373a4dc6" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:10.980953 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1ca1670916401ab0a43685df1b6759" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:10.992830 5592 ts_tablet_manager.cc:1918] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:11.000298 5592 log.cc:1198] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/961fef96e9124aafb1c51e287f603b64
I20250110 20:57:11.004001 5594 tablet_bootstrap.cc:492] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.004359 5594 ts_tablet_manager.cc:1397] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.215s user 0.009s sys 0.009s
I20250110 20:57:11.004277 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet fcc7b1acf4114bf69d801437855760df
I20250110 20:57:11.006111 4895 catalog_manager.cc:6596] ReplaceTablet: tablet fcc7b1acf4114bf69d801437855760df deleted and replaced by tablet 5efe92b10aea46608d55d5e5ac36eca1
I20250110 20:57:11.008839 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc7b1acf4114bf69d801437855760df with delete_type TABLET_DATA_DELETED (replaced by tablet 5efe92b10aea46608d55d5e5ac36eca1) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.003352 5595 ts_tablet_manager.cc:1918] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:11.010571 5592 ts_tablet_manager.cc:1939] T 961fef96e9124aafb1c51e287f603b64 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:11.011761 5595 log.cc:1198] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:11.016036 5595 ts_tablet_manager.cc:1939] T f0b5ba9f38c1493bba01180744f080e1 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:11.010947 5594 raft_consensus.cc:357] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.017992 5594 raft_consensus.cc:383] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.017962 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1ca1670916401ab0a43685df1b6759" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:11.018245 5594 raft_consensus.cc:738] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.018226 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet c813130649cc4df1b483ac4a86b4f0a0
I20250110 20:57:11.018783 5594 consensus_queue.cc:260] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:11.023749 5257 leader_election.cc:343] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.025880 5594 ts_tablet_manager.cc:1428] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20250110 20:57:11.026347 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.027940 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 131) took 1009 ms. Trace:
I20250110 20:57:11.029827 5592 rpcz_store.cc:276] 0110 20:57:10.018817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.086828 (+ 68011us) service_pool.cc:225] Handling call
0110 20:57:10.662735 (+575907us) ts_tablet_manager.cc:1145] Deleting tablet 961fef96e9124aafb1c51e287f603b64
0110 20:57:10.662773 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.808032 (+145259us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.992607 (+184575us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.027769 (+ 35162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":574924}
I20250110 20:57:11.026671 5594 tablet_bootstrap.cc:492] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:11.034662 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 961fef96e9124aafb1c51e287f603b64 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]) successfully deleted
W20250110 20:57:11.038024 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 2c1ca1670916401ab0a43685df1b6759 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c1ca1670916401ab0a43685df1b6759 already in progress: creating tablet
W20250110 20:57:11.041447 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet c6914e258314423c859eb31e85853274 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6914e258314423c859eb31e85853274 already in progress: creating tablet
I20250110 20:57:11.021371 5597 tablet_bootstrap.cc:654] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.047287 5594 tablet_bootstrap.cc:654] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:11.047755 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 2d634add82994f74960cd2b8373a4dc6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d634add82994f74960cd2b8373a4dc6 already in progress: creating tablet
I20250110 20:57:11.025723 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1fafed70adb496d97acfd5fea356292" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:11.029654 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.036310 5916 tablet.cc:1721] T 472174c6885b446a839721157914f047 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:11.036688 5173 raft_consensus.cc:2463] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:11.052927 5592 tablet_replica.cc:331] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:11.054210 5399 leader_election.cc:343] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:11.055083 5399 leader_election.cc:304] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:11.055738 5907 raft_consensus.cc:2743] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.056656 5399 leader_election.cc:304] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:11.057355 5907 raft_consensus.cc:2798] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:11.057770 5907 raft_consensus.cc:695] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:11.058538 5907 consensus_queue.cc:237] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.060267 5594 tablet_bootstrap.cc:492] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.060598 5594 ts_tablet_manager.cc:1397] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20250110 20:57:11.061856 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc7b1acf4114bf69d801437855760df with delete_type TABLET_DATA_DELETED (replaced by tablet 5efe92b10aea46608d55d5e5ac36eca1) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.064918 5594 raft_consensus.cc:357] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.056550 5592 raft_consensus.cc:2238] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:11.071724 5594 raft_consensus.cc:383] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.071961 5592 raft_consensus.cc:2267] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.072288 5594 raft_consensus.cc:738] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.074496 5594 consensus_queue.cc:260] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.077708 5592 ts_tablet_manager.cc:1905] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:11.080139 5594 ts_tablet_manager.cc:1428] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250110 20:57:11.084108 5594 tablet_bootstrap.cc:492] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:11.097052 5594 tablet_bootstrap.cc:654] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:10.943854 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d6eca349514b9fb0fff7cd6230027c" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
W20250110 20:57:11.103664 5398 leader_election.cc:343] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: d9776df98cb54be38a521852634490e7
I20250110 20:57:11.105995 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 120) took 2075 ms. Trace:
I20250110 20:57:11.106348 5595 rpcz_store.cc:276] 0110 20:57:09.030726 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.030837 (+ 111us) service_pool.cc:225] Handling call
0110 20:57:10.700360 (+1669523us) ts_tablet_manager.cc:1145] Deleting tablet f0b5ba9f38c1493bba01180744f080e1
0110 20:57:10.700395 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:10.929699 (+229304us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.002034 (+ 72335us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.105939 (+103905us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":83582,"tablet-delete.queue_time_us":1668990}
I20250110 20:57:11.107461 5595 tablet_replica.cc:331] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
W20250110 20:57:11.111864 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet cd0a666892d04e97a4b9967e38ab0946 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd0a666892d04e97a4b9967e38ab0946 already in progress: creating tablet
I20250110 20:57:11.124488 4896 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a809544a8ed648619ae05e19bdd40599" } new_table_name: "default.Table_85d9be9fb9414103a1623ee3dbe714ae" modify_external_catalogs: true
I20250110 20:57:10.953101 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae5956b1aa17449497f9695a111ebb66" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:11.132376 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 527bbddda8184a6eb67c2411ae50db63
I20250110 20:57:11.133340 5592 ts_tablet_manager.cc:1918] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:11.133671 4900 catalog_manager.cc:6596] ReplaceTablet: tablet 527bbddda8184a6eb67c2411ae50db63 deleted and replaced by tablet 9633eb4580c04f11a48ff5055e257c50
I20250110 20:57:11.133700 5592 log.cc:1198] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/f1fafed70adb496d97acfd5fea356292
I20250110 20:57:11.134227 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 527bbddda8184a6eb67c2411ae50db63 with delete_type TABLET_DATA_DELETED (replaced by tablet 9633eb4580c04f11a48ff5055e257c50) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.135133 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_cf6aeef756774db6a25af02ae4a44197"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:11.140865 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet 527bbddda8184a6eb67c2411ae50db63 with delete_type TABLET_DATA_DELETED (replaced by tablet 9633eb4580c04f11a48ff5055e257c50) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.141577 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_31bd404aec6f4df489bae22a80054bfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:11.142688 5592 ts_tablet_manager.cc:1939] T f1fafed70adb496d97acfd5fea356292 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
W20250110 20:57:11.142942 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31bd404aec6f4df489bae22a80054bfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:11.144657 5594 tablet_bootstrap.cc:492] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.145437 5594 ts_tablet_manager.cc:1397] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.061s user 0.015s sys 0.004s
I20250110 20:57:11.146575 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet f1fafed70adb496d97acfd5fea356292 (table default.Table_5107a94b926b4518baaf9bacd04d2f46 [id=e2dbf27ca6424662a7a14c9a9ae5587d]) successfully deleted
W20250110 20:57:11.147202 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet fcc7b1acf4114bf69d801437855760df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcc7b1acf4114bf69d801437855760df already in progress: creating tablet
W20250110 20:57:11.147719 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet c813130649cc4df1b483ac4a86b4f0a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c813130649cc4df1b483ac4a86b4f0a0 already in progress: creating tablet
W20250110 20:57:11.148303 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 527bbddda8184a6eb67c2411ae50db63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 527bbddda8184a6eb67c2411ae50db63 already in progress: creating tablet
W20250110 20:57:11.137140 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf6aeef756774db6a25af02ae4a44197 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:11.104995 5880 raft_consensus.cc:491] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:11.156852 5594 raft_consensus.cc:357] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.157574 5594 raft_consensus.cc:383] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.157883 5594 raft_consensus.cc:738] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.110544 5323 raft_consensus.cc:3054] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:11.158560 5594 consensus_queue.cc:260] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.160142 5594 ts_tablet_manager.cc:1428] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20250110 20:57:11.160949 5594 tablet_bootstrap.cc:492] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:11.165831 5594 tablet_bootstrap.cc:654] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.155030 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c813130649cc4df1b483ac4a86b4f0a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:11.207403 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 491626848ad54be3a18186c65d6dbb60
I20250110 20:57:11.209127 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 491626848ad54be3a18186c65d6dbb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.210662 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet 491626848ad54be3a18186c65d6dbb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:11.212085 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 491626848ad54be3a18186c65d6dbb60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 491626848ad54be3a18186c65d6dbb60 already in progress: creating tablet
I20250110 20:57:11.107246 5321 raft_consensus.cc:2463] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
W20250110 20:57:11.218803 5600 thread.cc:635] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.081s sys 0.176s
W20250110 20:57:11.152469 5255 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 44621us
W20250110 20:57:11.219527 5600 thread.cc:601] log-alloc [worker] (thread pool) Time spent starting thread: real 0.537s user 0.081s sys 0.176s
I20250110 20:57:11.222388 5319 raft_consensus.cc:2388] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in current term 1: Already voted for candidate ac9003be04f340f3886b5625cc928eb0 in this term.
I20250110 20:57:11.228045 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b89da7c5e86341c39853bbacce851ad4" } modify_external_catalogs: true
I20250110 20:57:11.228597 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b89da7c5e86341c39853bbacce851ad4" } modify_external_catalogs: true
I20250110 20:57:11.232455 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_8d629d1d2f524cd6964051cb0c0d2841"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:11.233913 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d629d1d2f524cd6964051cb0c0d2841 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:11.159695 5880 raft_consensus.cc:513] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.238868 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bae52ea080f54d228b6b76922248ed5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:11.158386 5297 tablet_service.cc:1467] Processing CreateTablet for tablet fb8c87076a8a4b1e86f7e4844e17b1f3 (DEFAULT_TABLE table=default.Table_368911cd6dd24875993ac8c44548a12d [id=e6b770e622ab4e3db1a8927d3d6894be]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:11.239758 5918 raft_consensus.cc:491] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:11.246899 5929 raft_consensus.cc:491] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:11.247339 5929 raft_consensus.cc:513] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.250350 5929 leader_election.cc:290] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:11.226584 5255 leader_election.cc:343] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.253994 5255 leader_election.cc:304] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:11.258025 5110 consensus_peers.cc:487] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 961fef96e9124aafb1c51e287f603b64. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:11.259168 5600 tablet_bootstrap.cc:492] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:11.259560 5600 ts_tablet_manager.cc:1397] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.612s user 0.085s sys 0.181s
I20250110 20:57:11.260866 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet f0b5ba9f38c1493bba01180744f080e1 (table default.Table_5356ef036bd048488ffadd8d4b4caf58 [id=cdc76f7bfc594bb8bc9f56ccca1a2155]) successfully deleted
I20250110 20:57:11.261698 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8c87076a8a4b1e86f7e4844e17b1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:11.265710 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eb75e1886e74a719dc09ccad8e1e284. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:11.266520 5880 leader_election.cc:290] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:11.267213 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c57941aee12a4ebea2b47f4727fc6eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:11.258550 5918 raft_consensus.cc:513] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.279901 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1afe41a09954fc8873ea53a330ea03f" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:11.280084 5299 tablet_service.cc:1514] Processing DeleteTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:11.280663 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet 527bbddda8184a6eb67c2411ae50db63 with delete_type TABLET_DATA_DELETED (replaced by tablet 9633eb4580c04f11a48ff5055e257c50) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:11.281214 5299 tablet_service.cc:1514] Processing DeleteTablet for tablet 491626848ad54be3a18186c65d6dbb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:11.281764 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527bbddda8184a6eb67c2411ae50db63" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:11.282948 5255 leader_election.cc:343] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:11.283627 5255 leader_election.cc:343] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.284168 5595 raft_consensus.cc:2238] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:11.299139 5827 raft_consensus.cc:491] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:11.299553 5827 raft_consensus.cc:513] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.260888 5929 raft_consensus.cc:2743] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.304168 5827 leader_election.cc:290] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:11.305714 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac07ac6fc9f477ba7f9accc5533be39" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:11.306298 5458 raft_consensus.cc:2463] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:11.307451 5110 leader_election.cc:304] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:11.311211 5595 raft_consensus.cc:2267] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.312309 5827 raft_consensus.cc:2798] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:11.312690 5827 raft_consensus.cc:491] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:11.313103 5827 raft_consensus.cc:3054] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:11.315150 5595 ts_tablet_manager.cc:1905] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:11.317732 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951d5ca47e8409db646b43bb04a7034" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:11.318037 5600 raft_consensus.cc:357] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.318853 5600 raft_consensus.cc:383] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:11.318892 5255 leader_election.cc:343] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.319186 5600 raft_consensus.cc:738] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.319396 5918 leader_election.cc:290] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:11.319905 5600 consensus_queue.cc:260] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.328413 5458 raft_consensus.cc:3054] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:11.332149 5323 raft_consensus.cc:2463] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:11.347312 5296 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc7b1acf4114bf69d801437855760df with delete_type TABLET_DATA_DELETED (replaced by tablet 5efe92b10aea46608d55d5e5ac36eca1) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:11.353721 5887 consensus_queue.cc:1035] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:11.322067 5600 ts_tablet_manager.cc:1428] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.062s user 0.003s sys 0.004s
I20250110 20:57:11.378244 5600 tablet_bootstrap.cc:492] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:11.381484 5111 leader_election.cc:304] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6
I20250110 20:57:11.390463 5398 leader_election.cc:304] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:11.391587 5908 raft_consensus.cc:2743] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.392920 5773 ts_tablet_manager.cc:1918] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:11.394634 5773 log.cc:1198] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/f1fafed70adb496d97acfd5fea356292
I20250110 20:57:11.395538 5908 raft_consensus.cc:491] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:11.395972 5908 raft_consensus.cc:513] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.397825 5908 leader_election.cc:290] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:11.407670 5398 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:11.410382 5924 tablet.cc:1721] T 472174c6885b446a839721157914f047 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:11.489629 5773 ts_tablet_manager.cc:1939] T f1fafed70adb496d97acfd5fea356292 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:11.490602 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 115) took 2323 ms. Trace:
I20250110 20:57:11.484107 5597 tablet_bootstrap.cc:492] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:11.491856 5597 ts_tablet_manager.cc:1397] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.605s user 0.007s sys 0.009s
I20250110 20:57:11.490998 5773 rpcz_store.cc:276] 0110 20:57:09.167596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.226061 (+ 58465us) service_pool.cc:225] Handling call
0110 20:57:10.803286 (+1577225us) ts_tablet_manager.cc:1145] Deleting tablet f1fafed70adb496d97acfd5fea356292
0110 20:57:10.803329 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.042511 (+239182us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.392681 (+350170us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.490546 (+ 97865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576706}
I20250110 20:57:11.492949 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 121) took 1826 ms. Trace:
W20250110 20:57:11.502655 5920 log.cc:926] Time spent T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20250110 20:57:11.503908 5597 raft_consensus.cc:357] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.504509 5597 raft_consensus.cc:383] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.504745 5597 raft_consensus.cc:738] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.505254 5597 consensus_queue.cc:260] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.507143 5597 ts_tablet_manager.cc:1428] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250110 20:57:11.511868 5600 tablet_bootstrap.cc:654] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.512449 5581 webserver.cc:458] Webserver started at http://127.4.171.252:45215/ using document root <none> and password file <none>
I20250110 20:57:11.513805 5581 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:11.514149 5581 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250110 20:57:11.516860 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 135487us
I20250110 20:57:11.518218 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b65488c884452d876ced1c3d1efb9a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:11.518792 5173 raft_consensus.cc:2463] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
W20250110 20:57:11.519857 5111 consensus_peers.cc:487] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 477b09061a4a4f278720168872a83258. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:11.521770 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 108641us
I20250110 20:57:11.524317 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet f1fafed70adb496d97acfd5fea356292 (table default.Table_5107a94b926b4518baaf9bacd04d2f46 [id=e2dbf27ca6424662a7a14c9a9ae5587d]) successfully deleted
I20250110 20:57:11.494971 5827 raft_consensus.cc:513] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.536787 5827 leader_election.cc:290] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:11.538641 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]). Expected version 2 got 0
I20250110 20:57:11.539196 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:11.542797 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
W20250110 20:57:11.556622 4894 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 46) took 841 ms (client timeout 1000 ms). Trace:
I20250110 20:57:11.557816 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.558436 5592 tablet_replica.cc:331] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:11.559109 5592 raft_consensus.cc:2238] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
W20250110 20:57:11.557057 4894 rpcz_store.cc:269] 0110 20:57:10.714883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.720699 (+ 5816us) service_pool.cc:225] Handling call
0110 20:57:10.877879 (+157180us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:11.556548 (+678669us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":358327552}
I20250110 20:57:11.559711 5592 pending_rounds.cc:62] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:11.559926 5592 pending_rounds.cc:69] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878616825532416 op_type: NO_OP noop_request { }
I20250110 20:57:11.560274 5592 raft_consensus.cc:2883] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:11.560570 5592 raft_consensus.cc:2267] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.560521 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]). Expected version 2 got 0
I20250110 20:57:11.495612 5773 rpcz_store.cc:276] 0110 20:57:09.666747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.688442 (+ 21695us) service_pool.cc:225] Handling call
0110 20:57:11.492639 (+1804197us) ts_tablet_manager.cc:1145] Deleting tablet 637a39e4264b40dd9480dc13e37359d9
0110 20:57:11.492676 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.492903 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1803525}
I20250110 20:57:11.571115 5592 ts_tablet_manager.cc:1905] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:11.575394 4899 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 37) took 1140 ms (client timeout 1000 ms). Trace:
I20250110 20:57:11.577989 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 130) took 1304 ms. Trace:
I20250110 20:57:11.578281 5773 rpcz_store.cc:276] 0110 20:57:10.273359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.273594 (+ 235us) service_pool.cc:225] Handling call
0110 20:57:11.566969 (+1293375us) ts_tablet_manager.cc:1145] Deleting tablet c6914e258314423c859eb31e85853274
0110 20:57:11.567007 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.577936 (+ 10929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292899}
I20250110 20:57:11.579321 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 129) took 1306 ms. Trace:
I20250110 20:57:11.579602 5773 rpcz_store.cc:276] 0110 20:57:10.273080 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.274184 (+ 1104us) service_pool.cc:225] Handling call
0110 20:57:11.579032 (+1304848us) ts_tablet_manager.cc:1145] Deleting tablet 2c1ca1670916401ab0a43685df1b6759
0110 20:57:11.579067 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.579275 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1304400}
I20250110 20:57:11.580523 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 133) took 1228 ms. Trace:
I20250110 20:57:11.580796 5773 rpcz_store.cc:276] 0110 20:57:10.351933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.354264 (+ 2331us) service_pool.cc:225] Handling call
0110 20:57:11.580318 (+1226054us) ts_tablet_manager.cc:1145] Deleting tablet 8fe93f1a08f3432dba9c6ce76b4c99b0
0110 20:57:11.580350 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.580480 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1225515}
I20250110 20:57:11.583657 5592 ts_tablet_manager.cc:1918] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:11.583982 5592 log.cc:1198] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/d9776df98cb54be38a521852634490e7
I20250110 20:57:11.584612 5592 ts_tablet_manager.cc:1939] T d9776df98cb54be38a521852634490e7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:11.558792 5918 raft_consensus.cc:491] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:11.585147 5918 raft_consensus.cc:513] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:11.591295 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 637a39e4264b40dd9480dc13e37359d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 637a39e4264b40dd9480dc13e37359d9 already in progress: creating tablet
W20250110 20:57:11.592276 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet c6914e258314423c859eb31e85853274 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6914e258314423c859eb31e85853274 already in progress: creating tablet
W20250110 20:57:11.593093 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 2c1ca1670916401ab0a43685df1b6759 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c1ca1670916401ab0a43685df1b6759 already in progress: creating tablet
I20250110 20:57:11.598349 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]). Expected version 2 got 0
I20250110 20:57:11.598886 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:11.599251 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet d9776df98cb54be38a521852634490e7 (table default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f [id=3bbd1acd709b44899907148df944b20d]) successfully deleted
I20250110 20:57:11.599437 4894 catalog_manager.cc:5526] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to ac9003be04f340f3886b5625cc928eb0 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "ac9003be04f340f3886b5625cc928eb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } }
I20250110 20:57:11.600515 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet ae5956b1aa17449497f9695a111ebb66 (table default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]). Expected version 1 got 0
W20250110 20:57:11.601929 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 8fe93f1a08f3432dba9c6ce76b4c99b0 because the tablet was not found. No further retry: Not found: Tablet not found: 8fe93f1a08f3432dba9c6ce76b4c99b0
I20250110 20:57:11.603494 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 961fef96e9124aafb1c51e287f603b64 with delete_type TABLET_DATA_DELETED (replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.604421 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.605233 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 with delete_type TABLET_DATA_DELETED (replaced by tablet 48e53d95ef8645c0bd123138655305ac) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.606060 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.516978 5936 raft_consensus.cc:2798] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:11.608264 5936 raft_consensus.cc:491] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:11.608579 5936 raft_consensus.cc:3054] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:11.575683 4899 rpcz_store.cc:269] 0110 20:57:10.434741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.434872 (+ 131us) service_pool.cc:225] Handling call
0110 20:57:10.457141 (+ 22269us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:11.575213 (+1118072us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:11.198155 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:11.501125 (+302970us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:11.501150 (+ 25us) write_op.cc:435] Acquired schema lock
0110 20:57:11.501163 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:11.501431 (+ 268us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:11.501469 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:11.501490 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:11.501625 (+ 135us) tablet.cc:698] Row locks acquired
0110 20:57:11.501636 (+ 11us) write_op.cc:260] PREPARE: finished
0110 20:57:11.501742 (+ 106us) write_op.cc:270] Start()
0110 20:57:11.501820 (+ 78us) write_op.cc:276] Timestamp: P: 1736542631501715 usec, L: 0
0110 20:57:11.501845 (+ 25us) op_driver.cc:348] REPLICATION: starting
0110 20:57:11.502324 (+ 479us) log.cc:844] Serialized 533 byte log entry
0110 20:57:11.532032 (+ 29708us) op_driver.cc:464] REPLICATION: finished
0110 20:57:11.532450 (+ 418us) write_op.cc:301] APPLY: starting
0110 20:57:11.532506 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:11.532597 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:11.532686 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:11.532869 (+ 183us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:11.532995 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:11.533035 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:11.533207 (+ 172us) log.cc:844] Serialized 23 byte log entry
0110 20:57:11.533337 (+ 130us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:11.533388 (+ 51us) write_op.cc:454] Released schema lock
0110 20:57:11.533431 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:11.533632 (+ 201us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:11.533658 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":17020,"reported_tablets":11,"spinlock_wait_cycles":46478080,"child_traces":[["op",{"apply.queue_time_us":263,"apply.run_cpu_time_us":1161,"apply.run_wall_time_us":1162,"mutex_wait_us":22823,"num_ops":1,"prepare.queue_time_us":438759,"prepare.run_cpu_time_us":2066,"prepare.run_wall_time_us":330944,"raft.queue_time_us":23069,"raft.run_cpu_time_us":671,"raft.run_wall_time_us":673,"replication_time_us":30096}]]}
I20250110 20:57:11.616919 5597 tablet_bootstrap.cc:492] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:11.621141 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 100209us
W20250110 20:57:11.621747 5111 leader_election.cc:343] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:11.622653 5399 leader_election.cc:304] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:11.623544 5907 raft_consensus.cc:2798] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:11.623917 5907 raft_consensus.cc:491] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:11.624275 5907 raft_consensus.cc:3054] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:11.624930 5773 tablet_replica.cc:331] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:11.625648 5773 raft_consensus.cc:2238] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:11.626358 5773 pending_rounds.cc:62] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:11.626571 5773 pending_rounds.cc:69] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878611442073600 op_type: NO_OP noop_request { }
I20250110 20:57:11.626890 5773 raft_consensus.cc:2883] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:11.627183 5773 raft_consensus.cc:2267] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.628811 5773 ts_tablet_manager.cc:1905] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:11.631300 5907 raft_consensus.cc:513] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.632550 5595 ts_tablet_manager.cc:1918] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:11.632964 5595 log.cc:1198] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/da24904dece04a6194790054f6d7153a
I20250110 20:57:11.633524 5907 leader_election.cc:290] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:11.634094 5595 ts_tablet_manager.cc:1939] T da24904dece04a6194790054f6d7153a P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:11.635914 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1afe41a09954fc8873ea53a330ea03f" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:11.637279 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527bbddda8184a6eb67c2411ae50db63" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:11.638522 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951d5ca47e8409db646b43bb04a7034" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:11.644810 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c951d5ca47e8409db646b43bb04a7034 consensus state has no leader
I20250110 20:57:11.645926 4893 master_service.cc:946] ReplaceTablet: received request to replace tablet c951d5ca47e8409db646b43bb04a7034 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:11.650869 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ac07ac6fc9f477ba7f9accc5533be39" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:11.651402 5458 raft_consensus.cc:3054] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:11.651907 4891 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 53) took 914 ms (client timeout 1000 ms). Trace:
W20250110 20:57:11.652176 4891 rpcz_store.cc:269] 0110 20:57:10.737744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.766637 (+ 28893us) service_pool.cc:225] Handling call
0110 20:57:10.871891 (+105254us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:11.651827 (+779936us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17123,"reported_tablets":6,"spinlock_wait_cycles":236868480}
I20250110 20:57:11.658869 5458 raft_consensus.cc:2463] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:11.660878 4899 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_931826e8e40a4d04a1382a2816608f99" } modify_external_catalogs: true
W20250110 20:57:11.674036 5398 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:11.675912 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:11.678689 5594 tablet_bootstrap.cc:492] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.680063 5594 ts_tablet_manager.cc:1397] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.519s user 0.133s sys 0.237s
I20250110 20:57:11.672344 5918 leader_election.cc:290] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:11.677551 5255 leader_election.cc:343] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:11.678282 5257 leader_election.cc:343] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.680758 5257 leader_election.cc:304] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:11.681620 5918 raft_consensus.cc:2743] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.681879 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 122) took 2648 ms. Trace:
W20250110 20:57:11.682487 5257 leader_election.cc:343] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.682149 5595 rpcz_store.cc:276] 0110 20:57:09.033690 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.033807 (+ 117us) service_pool.cc:225] Handling call
0110 20:57:11.107318 (+2073511us) ts_tablet_manager.cc:1145] Deleting tablet da24904dece04a6194790054f6d7153a
0110 20:57:11.107358 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.455288 (+347930us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.632344 (+177056us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.681817 (+ 49473us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":175898,"tablet-delete.queue_time_us":2073026}
I20250110 20:57:11.682833 5257 leader_election.cc:304] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:11.683274 5595 tablet_replica.cc:331] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:11.683941 5595 raft_consensus.cc:2238] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:11.684326 5595 raft_consensus.cc:2267] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:11.683662 5257 leader_election.cc:343] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.678084 4898 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36568 (request call id 180) took 1066 ms. Trace:
I20250110 20:57:11.684785 5257 leader_election.cc:304] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:11.686012 5929 raft_consensus.cc:2743] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.686493 5929 raft_consensus.cc:2743] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.685204 4898 rpcz_store.cc:276] 0110 20:57:10.611528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.706747 (+ 95219us) service_pool.cc:225] Handling call
0110 20:57:10.874856 (+168109us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b306 00007f3bfcfff044 00007f3bfcffefd2 00007f3bfcffef01 00007f3bfcffee7c 00007f3bfcffe132 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:10.876722 (+ 1866us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:10.881541 (+ 4819us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:11.677237 (+795696us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:11.677668 (+ 431us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:11.677938 (+ 270us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:11.640189 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:11.640536 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:11.640557 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:57:11.640569 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:11.640824 (+ 255us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:11.640868 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:11.640888 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:11.664201 (+ 23313us) tablet.cc:698] Row locks acquired
0110 20:57:11.664217 (+ 16us) write_op.cc:260] PREPARE: finished
0110 20:57:11.664314 (+ 97us) write_op.cc:270] Start()
0110 20:57:11.664391 (+ 77us) write_op.cc:276] Timestamp: P: 1736542631664267 usec, L: 0
0110 20:57:11.664404 (+ 13us) op_driver.cc:348] REPLICATION: starting
0110 20:57:11.664846 (+ 442us) log.cc:844] Serialized 433 byte log entry
0110 20:57:11.675067 (+ 10221us) op_driver.cc:464] REPLICATION: finished
0110 20:57:11.675409 (+ 342us) write_op.cc:301] APPLY: starting
0110 20:57:11.675459 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:11.675553 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:11.675643 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:11.676013 (+ 370us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:11.676137 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:11.676175 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:57:11.676331 (+ 156us) log.cc:844] Serialized 29 byte log entry
0110 20:57:11.676455 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:11.676517 (+ 62us) write_op.cc:454] Released schema lock
0110 20:57:11.676544 (+ 27us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":384915328,"child_traces":[["op",{"apply.queue_time_us":212,"apply.run_cpu_time_us":1296,"apply.run_wall_time_us":1297,"num_ops":2,"prepare.queue_time_us":757621,"raft.queue_time_us":77,"raft.run_cpu_time_us":589,"raft.run_wall_time_us":590,"replication_time_us":10586}]]}
I20250110 20:57:11.688318 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet da24904dece04a6194790054f6d7153a (table default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]) successfully deleted
I20250110 20:57:11.689231 4899 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_931826e8e40a4d04a1382a2816608f99" } modify_external_catalogs: true
I20250110 20:57:11.695331 5594 raft_consensus.cc:357] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.694094 4898 master_service.cc:946] ReplaceTablet: received request to replace tablet 31b65488c884452d876ced1c3d1efb9a from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:11.695930 5594 raft_consensus.cc:383] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.696169 5594 raft_consensus.cc:738] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.696684 5594 consensus_queue.cc:260] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.705466 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31b65488c884452d876ced1c3d1efb9a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:11.705984 5174 raft_consensus.cc:3054] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:11.709722 5594 ts_tablet_manager.cc:1428] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.018s user 0.004s sys 0.001s
W20250110 20:57:11.709781 4896 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d5c46de95bc4227bffc5e8425b8cc95 consensus state has no leader
I20250110 20:57:11.714924 5594 tablet_bootstrap.cc:492] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:11.717552 5174 raft_consensus.cc:2463] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:11.718812 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:11.719674 5594 tablet_bootstrap.cc:654] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.720275 5595 ts_tablet_manager.cc:1905] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:11.729292 5111 consensus_peers.cc:487] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 961fef96e9124aafb1c51e287f603b64. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:11.729961 5111 leader_election.cc:343] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:11.732199 5942 consensus_queue.cc:1035] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:11.734283 5110 leader_election.cc:304] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:11.735030 5890 raft_consensus.cc:2798] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:11.735428 5890 raft_consensus.cc:695] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:11.736166 5890 consensus_queue.cc:237] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.744169 5597 tablet_bootstrap.cc:654] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.744746 5594 tablet_bootstrap.cc:492] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.745095 5594 ts_tablet_manager.cc:1397] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.003s
I20250110 20:57:11.749771 5594 raft_consensus.cc:357] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.750391 5594 raft_consensus.cc:383] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.750620 5594 raft_consensus.cc:738] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.751127 5594 consensus_queue.cc:260] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.752822 5594 ts_tablet_manager.cc:1428] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250110 20:57:11.756904 5594 tablet_bootstrap.cc:492] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
W20250110 20:57:11.767800 5257 leader_election.cc:343] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.768188 5257 leader_election.cc:304] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:11.769505 5880 raft_consensus.cc:2743] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.770646 5594 tablet_bootstrap.cc:654] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.781826 5399 leader_election.cc:304] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:11.782871 5907 raft_consensus.cc:2798] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:11.783751 5600 tablet_bootstrap.cc:492] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:11.783821 5907 raft_consensus.cc:695] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:11.784142 5600 ts_tablet_manager.cc:1397] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.406s user 0.015s sys 0.016s
I20250110 20:57:11.786563 5600 raft_consensus.cc:357] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.787181 5600 raft_consensus.cc:383] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.787416 5600 raft_consensus.cc:738] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.784898 5907 consensus_queue.cc:237] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.788004 5600 consensus_queue.cc:260] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.793644 5600 ts_tablet_manager.cc:1428] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250110 20:57:11.797868 5600 tablet_bootstrap.cc:492] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:11.808964 5600 tablet_bootstrap.cc:654] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.813184 5594 tablet_bootstrap.cc:492] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.813587 5594 ts_tablet_manager.cc:1397] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.001s
I20250110 20:57:11.749241 5936 raft_consensus.cc:513] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.772356 5773 ts_tablet_manager.cc:1918] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:11.815222 5773 log.cc:1198] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/477b09061a4a4f278720168872a83258
I20250110 20:57:11.815927 5773 ts_tablet_manager.cc:1939] T 477b09061a4a4f278720168872a83258 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:11.816617 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 135) took 1464 ms. Trace:
I20250110 20:57:11.816763 5953 tablet.cc:1721] T 472174c6885b446a839721157914f047 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:11.816905 5773 rpcz_store.cc:276] 0110 20:57:10.352397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.354954 (+ 2557us) service_pool.cc:225] Handling call
0110 20:57:11.624783 (+1269829us) ts_tablet_manager.cc:1145] Deleting tablet 477b09061a4a4f278720168872a83258
0110 20:57:11.624823 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.734626 (+109803us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.772160 (+ 37534us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.816567 (+ 44407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1269313}
I20250110 20:57:11.817898 5773 tablet_replica.cc:331] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:11.818404 5936 leader_election.cc:290] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:11.819613 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d634add82994f74960cd2b8373a4dc6" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:11.820201 5459 raft_consensus.cc:3054] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:11.820482 5594 raft_consensus.cc:357] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.821091 5594 raft_consensus.cc:383] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.821327 5594 raft_consensus.cc:738] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.821863 5594 consensus_queue.cc:260] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.824050 5594 ts_tablet_manager.cc:1428] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20250110 20:57:11.828883 5594 tablet_bootstrap.cc:492] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:11.818583 5773 raft_consensus.cc:2238] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:11.832353 5890 consensus_queue.cc:1035] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:11.839314 5594 tablet_bootstrap.cc:654] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.839702 5773 raft_consensus.cc:2267] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.841436 5773 ts_tablet_manager.cc:1905] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:11.847769 5594 tablet_bootstrap.cc:492] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.848126 5594 ts_tablet_manager.cc:1397] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20250110 20:57:11.850308 5594 raft_consensus.cc:357] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.850950 5594 raft_consensus.cc:383] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.851198 5594 raft_consensus.cc:738] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.827028 5459 raft_consensus.cc:2463] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:11.856139 5594 consensus_queue.cc:260] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:11.859221 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.252:37233, user_credentials={real_user=slave}} blocked reactor thread for 49239.8us
I20250110 20:57:11.860092 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 477b09061a4a4f278720168872a83258 (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]) successfully deleted
I20250110 20:57:11.860473 5594 ts_tablet_manager.cc:1428] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250110 20:57:11.853106 5110 leader_election.cc:304] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:11.861250 5594 tablet_bootstrap.cc:492] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
W20250110 20:57:11.861690 5110 raft_consensus.cc:2681] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250110 20:57:11.850845 5597 tablet_bootstrap.cc:492] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:11.865684 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 5c5b954dbccb45b2a6ab7cc5cceadffb
I20250110 20:57:11.869575 5597 ts_tablet_manager.cc:1397] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.253s user 0.008s sys 0.003s
I20250110 20:57:11.872120 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:11.873498 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c5b954dbccb45b2a6ab7cc5cceadffb already in progress: creating tablet
I20250110 20:57:11.873791 5581 fs_manager.cc:714] Time spent opening directory manager: real 0.278s user 0.023s sys 0.042s
I20250110 20:57:11.881630 4893 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet c951d5ca47e8409db646b43bb04a7034
I20250110 20:57:11.882850 4893 catalog_manager.cc:6596] ReplaceTablet: tablet c951d5ca47e8409db646b43bb04a7034 deleted and replaced by tablet 95b52e115a8e4bae95da0ef9d5afff8c
I20250110 20:57:11.884824 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet c951d5ca47e8409db646b43bb04a7034 with delete_type TABLET_DATA_DELETED (replaced by tablet 95b52e115a8e4bae95da0ef9d5afff8c) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.871655 5597 raft_consensus.cc:357] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:11.885228 5597 raft_consensus.cc:383] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.885488 5597 raft_consensus.cc:738] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.886042 5597 consensus_queue.cc:260] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:11.886957 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet c951d5ca47e8409db646b43bb04a7034 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c951d5ca47e8409db646b43bb04a7034 already in progress: creating tablet
I20250110 20:57:11.891232 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_156534e2fa0e4fa7bd3eb498d05a4626"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:11.892522 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_156534e2fa0e4fa7bd3eb498d05a4626 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:11.895298 5597 ts_tablet_manager.cc:1428] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20250110 20:57:11.902144 5597 tablet_bootstrap.cc:492] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:11.901962 4892 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1c8384b6009d43a89936a5f543415675" } new_table_name: "default.Table_f4762b68ddab45a685e0d06a7703ea3c" modify_external_catalogs: true
W20250110 20:57:11.903921 4885 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2d5c46de95bc4227bffc5e8425b8cc95 consensus state has no leader
I20250110 20:57:11.898629 5773 ts_tablet_manager.cc:1918] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:11.905244 5773 log.cc:1198] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/2d634add82994f74960cd2b8373a4dc6
I20250110 20:57:11.905848 5773 ts_tablet_manager.cc:1939] T 2d634add82994f74960cd2b8373a4dc6 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:11.906889 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 131) took 1555 ms. Trace:
I20250110 20:57:11.907359 5594 tablet_bootstrap.cc:654] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.907158 5773 rpcz_store.cc:276] 0110 20:57:10.351382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.355527 (+ 4145us) service_pool.cc:225] Handling call
0110 20:57:11.817760 (+1462233us) ts_tablet_manager.cc:1145] Deleting tablet 2d634add82994f74960cd2b8373a4dc6
0110 20:57:11.817796 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.867303 (+ 49507us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.898438 (+ 31135us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.906839 (+ 8401us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1429,"tablet-delete.queue_time_us":1461671}
I20250110 20:57:11.912748 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8d629d1d2f524cd6964051cb0c0d2841" } new_table_name: "default.Table_57c42d638822484caa513bd1ce8616d9" modify_external_catalogs: true
I20250110 20:57:11.920293 5597 tablet_bootstrap.cc:654] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:11.908272 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 134) took 1556 ms. Trace:
I20250110 20:57:11.926054 5961 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:11.821367 5595 ts_tablet_manager.cc:1918] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:11.932700 5595 log.cc:1198] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/e361fb8b47d74548b5a2326d25a9e28c
I20250110 20:57:11.932826 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.933334 5595 ts_tablet_manager.cc:1939] T e361fb8b47d74548b5a2326d25a9e28c P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:11.935343 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 123) took 2901 ms. Trace:
I20250110 20:57:11.935649 5595 rpcz_store.cc:276] 0110 20:57:09.033973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.034370 (+ 397us) service_pool.cc:225] Handling call
0110 20:57:11.683151 (+2648781us) ts_tablet_manager.cc:1145] Deleting tablet e361fb8b47d74548b5a2326d25a9e28c
0110 20:57:11.683185 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.774308 (+ 91123us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.821168 (+ 46860us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:11.935280 (+114112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2648288}
I20250110 20:57:11.936456 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet c951d5ca47e8409db646b43bb04a7034 with delete_type TABLET_DATA_DELETED (replaced by tablet 95b52e115a8e4bae95da0ef9d5afff8c) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:11.936658 5595 tablet_replica.cc:331] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:11.938612 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 2d634add82994f74960cd2b8373a4dc6 (table default.Table_ba448520fdb24cd1be60910f20647848 [id=505b9ca1a68e401aa6f6f28a41f09181]) successfully deleted
I20250110 20:57:11.939241 5929 raft_consensus.cc:491] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:11.939689 5929 raft_consensus.cc:513] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:11.941532 5594 tablet_bootstrap.cc:492] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:11.941907 5594 ts_tablet_manager.cc:1397] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.002s
I20250110 20:57:11.944000 5594 raft_consensus.cc:357] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:11.944612 5594 raft_consensus.cc:383] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:11.944854 5594 raft_consensus.cc:738] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:11.945190 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7f09df8e274f3e9542292d5c1ec6c6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:11.946445 4887 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 -> Peer 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233): Couldn't send request to peer 14d1e2d5c72f498f915c367e8750361f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250110 20:57:11.948292 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d634add82994f74960cd2b8373a4dc6" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:11.948843 5320 raft_consensus.cc:3054] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:11.949643 5890 consensus_queue.cc:1035] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:11.950893 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:11.951447 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet c951d5ca47e8409db646b43bb04a7034 with delete_type TABLET_DATA_DELETED (replaced by tablet 95b52e115a8e4bae95da0ef9d5afff8c) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:11.951985 5594 consensus_queue.cc:260] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:11.957019 5320 raft_consensus.cc:2463] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
W20250110 20:57:11.960070 4892 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5eb75e1886e74a719dc09ccad8e1e284 consensus state has no leader
I20250110 20:57:11.962294 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:11.964520 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet d9776df98cb54be38a521852634490e7 because the tablet was not found. No further retry: Not found: Tablet not found: d9776df98cb54be38a521852634490e7
I20250110 20:57:11.966854 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet 78538b2e7bcc45c7afaa145bad6e5813 with delete_type TABLET_DATA_DELETED (replaced by tablet 48e53d95ef8645c0bd123138655305ac) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.968093 5592 tablet_replica.cc:331] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:11.968755 5592 raft_consensus.cc:2238] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:11.968823 5434 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 with delete_type TABLET_DATA_DELETED (replaced by tablet 371fb7408f0149c8b00659e09ebaf48a) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.969203 5592 raft_consensus.cc:2267] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.969390 5594 ts_tablet_manager.cc:1428] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250110 20:57:11.969785 5434 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d634add82994f74960cd2b8373a4dc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 33db8a2fdd9c4fde985345604fd04215) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.970731 5592 ts_tablet_manager.cc:1905] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:11.972954 5257 leader_election.cc:343] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.972981 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b65488c884452d876ced1c3d1efb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:11.973150 5594 tablet_bootstrap.cc:492] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:11.937361 5595 raft_consensus.cc:2238] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:11.975317 5595 raft_consensus.cc:2267] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.976923 5595 ts_tablet_manager.cc:1905] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:11.924803 5773 rpcz_store.cc:276] 0110 20:57:10.352168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.356291 (+ 4123us) service_pool.cc:225] Handling call
0110 20:57:11.908042 (+1551751us) ts_tablet_manager.cc:1145] Deleting tablet a7e7ffb063594128ba137f9aea86bf15
0110 20:57:11.908078 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.908227 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1551245}
I20250110 20:57:11.978793 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 132) took 1627 ms. Trace:
I20250110 20:57:11.979087 5773 rpcz_store.cc:276] 0110 20:57:10.351664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.357333 (+ 5669us) service_pool.cc:225] Handling call
0110 20:57:11.978491 (+1621158us) ts_tablet_manager.cc:1145] Deleting tablet cd0a666892d04e97a4b9967e38ab0946
0110 20:57:11.978530 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:11.978745 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620638}
I20250110 20:57:11.981930 5929 leader_election.cc:290] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:11.982831 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7f09df8e274f3e9542292d5c1ec6c6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:11.983150 5594 tablet_bootstrap.cc:654] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:11.984210 5255 leader_election.cc:343] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:11.984591 5255 leader_election.cc:304] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:11.985674 5929 raft_consensus.cc:2743] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:11.987699 5773 tablet_replica.cc:331] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:11.988368 5773 raft_consensus.cc:2238] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:11.994139 5581 fs_manager.cc:730] Time spent opening block manager: real 0.118s user 0.005s sys 0.010s
W20250110 20:57:12.007431 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.254:34617, user_credentials={real_user=slave}} blocked reactor thread for 52355.3us
W20250110 20:57:12.008426 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet a7e7ffb063594128ba137f9aea86bf15 because the tablet was not found. No further retry: Not found: Tablet not found: a7e7ffb063594128ba137f9aea86bf15
W20250110 20:57:12.009105 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet cd0a666892d04e97a4b9967e38ab0946 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd0a666892d04e97a4b9967e38ab0946 already in progress: creating tablet
W20250110 20:57:12.009711 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet c813130649cc4df1b483ac4a86b4f0a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c813130649cc4df1b483ac4a86b4f0a0 already in progress: creating tablet
W20250110 20:57:12.010285 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet fcc7b1acf4114bf69d801437855760df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcc7b1acf4114bf69d801437855760df already in progress: creating tablet
W20250110 20:57:12.011174 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 527bbddda8184a6eb67c2411ae50db63 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 527bbddda8184a6eb67c2411ae50db63 already in progress: creating tablet
I20250110 20:57:12.015681 5773 pending_rounds.cc:62] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:12.016677 5773 pending_rounds.cc:69] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878614018760704 op_type: NO_OP noop_request { }
I20250110 20:57:12.017295 5773 raft_consensus.cc:2883] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:12.017776 5773 raft_consensus.cc:2267] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:11.994553 5581 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
uuid: "14d1e2d5c72f498f915c367e8750361f"
format_stamp: "Formatted at 2025-01-10 20:56:53 on dist-test-slave-98kh"
I20250110 20:57:12.019881 5581 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:12.020773 5594 tablet_bootstrap.cc:492] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.021135 5594 ts_tablet_manager.cc:1397] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s
I20250110 20:57:12.023113 5594 raft_consensus.cc:357] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.023703 5594 raft_consensus.cc:383] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.023943 5594 raft_consensus.cc:738] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.024456 5594 consensus_queue.cc:260] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.026393 5773 ts_tablet_manager.cc:1905] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:12.026975 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 491626848ad54be3a18186c65d6dbb60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 491626848ad54be3a18186c65d6dbb60 already in progress: creating tablet
I20250110 20:57:12.031855 5592 ts_tablet_manager.cc:1918] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:12.034654 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_3f8bafa0c19642adb6062c8156965b27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:12.036512 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ce0a88d3ee2c4c17996b244999136619"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:12.039815 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce0a88d3ee2c4c17996b244999136619 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:12.043550 5597 tablet_bootstrap.cc:492] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
W20250110 20:57:12.043589 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f8bafa0c19642adb6062c8156965b27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:12.043932 5597 ts_tablet_manager.cc:1397] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.142s user 0.007s sys 0.003s
I20250110 20:57:12.048861 5595 ts_tablet_manager.cc:1918] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:12.049223 5595 log.cc:1198] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:12.049815 5595 ts_tablet_manager.cc:1939] T 7cf0c87159574bfcbc7a34abe508f9d6 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:12.051483 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 126) took 2719 ms. Trace:
I20250110 20:57:12.051765 5595 rpcz_store.cc:276] 0110 20:57:09.331731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.331886 (+ 155us) service_pool.cc:225] Handling call
0110 20:57:11.936522 (+2604636us) ts_tablet_manager.cc:1145] Deleting tablet 7cf0c87159574bfcbc7a34abe508f9d6
0110 20:57:11.936559 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.022401 (+ 85842us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.048347 (+ 25946us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.051429 (+ 3082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2604114}
I20250110 20:57:12.052737 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 7cf0c87159574bfcbc7a34abe508f9d6 (table default.Table_fc5f72ec93dd40bbaeb1be5da336b098 [id=6cc26e56afd04d78abff401b7ae32d99]) successfully deleted
I20250110 20:57:12.063768 5595 tablet_replica.cc:331] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:12.064445 5595 raft_consensus.cc:2238] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:12.064836 5595 raft_consensus.cc:2267] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:12.066370 5595 ts_tablet_manager.cc:1905] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:12.070510 5597 raft_consensus.cc:357] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.071162 5597 raft_consensus.cc:383] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.071411 5597 raft_consensus.cc:738] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.071962 5597 consensus_queue.cc:260] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.072510 5600 tablet_bootstrap.cc:492] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:12.072898 5600 ts_tablet_manager.cc:1397] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.275s user 0.015s sys 0.010s
I20250110 20:57:12.077584 5600 raft_consensus.cc:357] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.080592 5600 raft_consensus.cc:383] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.081305 5600 raft_consensus.cc:738] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.082324 5600 consensus_queue.cc:260] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.083168 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a73e1fb622794aa4a8a231555b3d53ca" } modify_external_catalogs: true
I20250110 20:57:12.083644 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a73e1fb622794aa4a8a231555b3d53ca" } modify_external_catalogs: true
I20250110 20:57:12.085760 5600 ts_tablet_manager.cc:1428] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20250110 20:57:12.085362 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_054981f06f6f4bd4954b77b6b2ec4436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:12.086872 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_054981f06f6f4bd4954b77b6b2ec4436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:12.093484 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb535174548d496e920801723e0909b3 has no consensus state
I20250110 20:57:12.048277 5594 ts_tablet_manager.cc:1428] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250110 20:57:12.105590 5592 log.cc:1198] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/78538b2e7bcc45c7afaa145bad6e5813
I20250110 20:57:12.106204 5594 tablet_bootstrap.cc:492] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.107369 5597 ts_tablet_manager.cc:1428] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.063s user 0.006s sys 0.000s
I20250110 20:57:12.088613 5600 tablet_bootstrap.cc:492] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:12.108186 5597 tablet_bootstrap.cc:492] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:12.111192 5592 ts_tablet_manager.cc:1939] T 78538b2e7bcc45c7afaa145bad6e5813 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:12.112890 5594 tablet_bootstrap.cc:654] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.113631 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 78538b2e7bcc45c7afaa145bad6e5813 (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]) successfully deleted
I20250110 20:57:12.120504 5592 tablet_replica.cc:331] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:12.121157 5592 raft_consensus.cc:2238] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:12.121558 5592 raft_consensus.cc:2267] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:12.123077 5592 ts_tablet_manager.cc:1905] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:12.126235 5595 ts_tablet_manager.cc:1918] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:12.126657 5595 log.cc:1198] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:12.127282 5595 ts_tablet_manager.cc:1939] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:12.139637 5597 tablet_bootstrap.cc:654] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:12.145975 5964 log.cc:926] Time spent T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20250110 20:57:12.151962 5773 ts_tablet_manager.cc:1918] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:12.152323 5773 log.cc:1198] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/961fef96e9124aafb1c51e287f603b64
I20250110 20:57:12.152959 5773 ts_tablet_manager.cc:1939] T 961fef96e9124aafb1c51e287f603b64 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
W20250110 20:57:12.158346 4887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.252:37233: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20250110 20:57:12.162964 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ba448520fdb24cd1be60910f20647848" } new_table_name: "default.Table_094c089272f6475a989a44b5e1843fd5" modify_external_catalogs: true
I20250110 20:57:12.153101 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 127) took 2609 ms. Trace:
I20250110 20:57:12.175438 5595 rpcz_store.cc:276] 0110 20:57:09.543329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.543438 (+ 109us) service_pool.cc:225] Handling call
0110 20:57:12.063634 (+2520196us) ts_tablet_manager.cc:1145] Deleting tablet ba1b57de9e4b4b2ab6c3bce8fe78166a
0110 20:57:12.063668 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.093515 (+ 29847us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.126021 (+ 32506us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.153046 (+ 27025us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2519658}
I20250110 20:57:12.176548 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet ba1b57de9e4b4b2ab6c3bce8fe78166a (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
I20250110 20:57:12.177569 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 128) took 2578 ms. Trace:
I20250110 20:57:12.178050 4893 master_service.cc:946] ReplaceTablet: received request to replace tablet ae5956b1aa17449497f9695a111ebb66 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:12.177840 5595 rpcz_store.cc:276] 0110 20:57:09.599547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.660264 (+ 60717us) service_pool.cc:225] Handling call
0110 20:57:12.177236 (+2516972us) ts_tablet_manager.cc:1145] Deleting tablet 637a39e4264b40dd9480dc13e37359d9
0110 20:57:12.177276 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.177522 (+ 246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2515847}
W20250110 20:57:12.179517 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 637a39e4264b40dd9480dc13e37359d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 637a39e4264b40dd9480dc13e37359d9 already in progress: creating tablet
I20250110 20:57:12.180272 5595 tablet_replica.cc:331] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:12.180963 5595 raft_consensus.cc:2238] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:12.155444 5600 tablet_bootstrap.cc:654] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.191412 5594 tablet_bootstrap.cc:492] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.191788 5594 ts_tablet_manager.cc:1397] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.086s user 0.014s sys 0.004s
I20250110 20:57:12.193892 5594 raft_consensus.cc:357] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.194506 5594 raft_consensus.cc:383] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.194747 5594 raft_consensus.cc:738] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.195288 5594 consensus_queue.cc:260] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.181373 5595 raft_consensus.cc:2267] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:12.199601 5595 ts_tablet_manager.cc:1905] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:12.214536 5594 ts_tablet_manager.cc:1428] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20250110 20:57:12.218161 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 961fef96e9124aafb1c51e287f603b64 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]) successfully deleted
I20250110 20:57:12.218993 5773 tablet_replica.cc:331] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:12.225673 5773 raft_consensus.cc:2238] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:12.228062 5907 raft_consensus.cc:491] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.228278 5594 tablet_bootstrap.cc:492] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.228475 5907 raft_consensus.cc:513] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.231076 5907 leader_election.cc:290] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:12.234162 5398 leader_election.cc:343] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:12.241958 5600 tablet_bootstrap.cc:492] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:12.242337 5600 ts_tablet_manager.cc:1397] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.156s user 0.010s sys 0.005s
I20250110 20:57:12.244905 5600 raft_consensus.cc:357] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.245561 5600 raft_consensus.cc:383] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.245800 5600 raft_consensus.cc:738] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.246400 5600 consensus_queue.cc:260] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.248579 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 7df3aafea18f49249cceb8a9431064e6 (DEFAULT_TABLE table=default.Table_0b1d1f4267df44e6a2555f1080545d59 [id=7016ed2e98194e4c89d508e5f724cf04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.249864 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df3aafea18f49249cceb8a9431064e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.257092 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 7df3aafea18f49249cceb8a9431064e6 (DEFAULT_TABLE table=default.Table_0b1d1f4267df44e6a2555f1080545d59 [id=7016ed2e98194e4c89d508e5f724cf04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.260885 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df3aafea18f49249cceb8a9431064e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.262027 5600 ts_tablet_manager.cc:1428] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250110 20:57:12.262789 5600 tablet_bootstrap.cc:492] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:12.268021 5600 tablet_bootstrap.cc:654] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.272014 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 758430b311a8478c9cab2bcf79253b52 (DEFAULT_TABLE table=default.Table_70b6412a11364f1389adbac0db42cdb2 [id=73b603428841415b8d7fba6ef22ffe22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.272909 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758430b311a8478c9cab2bcf79253b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.275573 5434 tablet_service.cc:1467] Processing CreateTablet for tablet 758430b311a8478c9cab2bcf79253b52 (DEFAULT_TABLE table=default.Table_70b6412a11364f1389adbac0db42cdb2 [id=73b603428841415b8d7fba6ef22ffe22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.276554 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758430b311a8478c9cab2bcf79253b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.277606 5929 raft_consensus.cc:491] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.278019 5929 raft_consensus.cc:513] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.279786 5929 leader_election.cc:290] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.282534 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:12.283090 5459 raft_consensus.cc:2463] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:12.284415 5255 leader_election.cc:304] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:12.286441 5600 tablet_bootstrap.cc:492] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:12.286684 5929 raft_consensus.cc:2798] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.286801 5600 ts_tablet_manager.cc:1397] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.007s
I20250110 20:57:12.286998 5929 raft_consensus.cc:491] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.287292 5929 raft_consensus.cc:3054] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.288977 5600 raft_consensus.cc:357] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.289618 5600 raft_consensus.cc:383] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.289906 5600 raft_consensus.cc:738] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.290603 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 1bda90a78a5b4e2a839d496b9317f6c0 (DEFAULT_TABLE table=default.Table_16d52a446dab43b6929b4d8fcce5e5f1 [id=1577048ac45c4bdb816df8469ed87185]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.290585 5600 consensus_queue.cc:260] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.291615 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bda90a78a5b4e2a839d496b9317f6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.292851 5594 tablet_bootstrap.cc:654] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.296377 5600 ts_tablet_manager.cc:1428] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20250110 20:57:12.297191 5600 tablet_bootstrap.cc:492] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:12.303544 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 1bda90a78a5b4e2a839d496b9317f6c0 (DEFAULT_TABLE table=default.Table_16d52a446dab43b6929b4d8fcce5e5f1 [id=1577048ac45c4bdb816df8469ed87185]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.304503 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bda90a78a5b4e2a839d496b9317f6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.309612 5907 raft_consensus.cc:491] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.309881 5600 tablet_bootstrap.cc:654] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.310012 5907 raft_consensus.cc:513] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.320358 5907 leader_election.cc:290] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:12.321995 5398 leader_election.cc:343] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: da24904dece04a6194790054f6d7153a
I20250110 20:57:12.332836 5443 tablet_service.cc:1467] Processing CreateTablet for tablet a687769f69d844cc8564e61dc324d2b2 (DEFAULT_TABLE table=default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.333859 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a687769f69d844cc8564e61dc324d2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.336920 5595 ts_tablet_manager.cc:1918] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:12.337252 5595 log.cc:1198] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/c6914e258314423c859eb31e85853274
I20250110 20:57:12.337836 5595 ts_tablet_manager.cc:1939] T c6914e258314423c859eb31e85853274 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:12.343483 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 137) took 2065 ms. Trace:
I20250110 20:57:12.343757 5595 rpcz_store.cc:276] 0110 20:57:10.278285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.278393 (+ 108us) service_pool.cc:225] Handling call
0110 20:57:12.180143 (+1901750us) ts_tablet_manager.cc:1145] Deleting tablet c6914e258314423c859eb31e85853274
0110 20:57:12.180179 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.214744 (+ 34565us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.336675 (+121931us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.343430 (+ 6755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1901247}
I20250110 20:57:12.344825 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet c6914e258314423c859eb31e85853274 (table default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621]) successfully deleted
I20250110 20:57:12.346163 5292 tablet_service.cc:1467] Processing CreateTablet for tablet a687769f69d844cc8564e61dc324d2b2 (DEFAULT_TABLE table=default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.349578 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a687769f69d844cc8564e61dc324d2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.354763 5594 tablet_bootstrap.cc:492] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.355134 5594 ts_tablet_manager.cc:1397] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.127s user 0.001s sys 0.010s
I20250110 20:57:12.355283 5880 raft_consensus.cc:491] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.355700 5880 raft_consensus.cc:513] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.357357 5594 raft_consensus.cc:357] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.358009 5594 raft_consensus.cc:383] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.358294 5594 raft_consensus.cc:738] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.358816 5594 consensus_queue.cc:260] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.361796 5594 ts_tablet_manager.cc:1428] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250110 20:57:12.362537 5594 tablet_bootstrap.cc:492] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.370741 5595 tablet_replica.cc:331] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:12.233485 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a520230d9fd4a74914e46f784dcf747" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.378646 5173 raft_consensus.cc:2463] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.380019 5399 leader_election.cc:304] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:12.380949 5907 raft_consensus.cc:2798] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.381268 5907 raft_consensus.cc:491] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.381578 5907 raft_consensus.cc:3054] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.233882 5597 tablet_bootstrap.cc:492] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:12.258502 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 7df3aafea18f49249cceb8a9431064e6 (DEFAULT_TABLE table=default.Table_0b1d1f4267df44e6a2555f1080545d59 [id=7016ed2e98194e4c89d508e5f724cf04]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.384966 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7df3aafea18f49249cceb8a9431064e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.386004 5594 tablet_bootstrap.cc:654] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.386386 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:12.387668 5255 leader_election.cc:343] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.388386 5880 leader_election.cc:290] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:12.384099 5595 raft_consensus.cc:2238] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:12.389323 5595 raft_consensus.cc:2267] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:12.258754 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 758430b311a8478c9cab2bcf79253b52 (DEFAULT_TABLE table=default.Table_70b6412a11364f1389adbac0db42cdb2 [id=73b603428841415b8d7fba6ef22ffe22]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.390218 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 758430b311a8478c9cab2bcf79253b52. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.391090 5595 ts_tablet_manager.cc:1905] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:12.394191 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.273626 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 1bda90a78a5b4e2a839d496b9317f6c0 (DEFAULT_TABLE table=default.Table_16d52a446dab43b6929b4d8fcce5e5f1 [id=1577048ac45c4bdb816df8469ed87185]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.396224 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bda90a78a5b4e2a839d496b9317f6c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.281191 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.312798 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da24904dece04a6194790054f6d7153a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.402074 5172 raft_consensus.cc:2463] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.327455 5155 tablet_service.cc:1467] Processing CreateTablet for tablet a687769f69d844cc8564e61dc324d2b2 (DEFAULT_TABLE table=default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.403064 5399 leader_election.cc:304] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:12.403551 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a687769f69d844cc8564e61dc324d2b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.373376 5146 tablet_service.cc:1467] Processing CreateTablet for tablet 33db8a2fdd9c4fde985345604fd04215 (DEFAULT_TABLE table=default.Table_ba448520fdb24cd1be60910f20647848 [id=505b9ca1a68e401aa6f6f28a41f09181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.406626 5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33db8a2fdd9c4fde985345604fd04215. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:12.409945 5257 leader_election.cc:343] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.410318 5257 leader_election.cc:304] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:12.411279 5257 leader_election.cc:343] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.412122 5880 raft_consensus.cc:2743] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:12.420712 5977 raft_consensus.cc:2798] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.424762 5977 raft_consensus.cc:491] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.425146 5977 raft_consensus.cc:3054] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.430574 5594 tablet_bootstrap.cc:492] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.431634 5594 ts_tablet_manager.cc:1397] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.001s
I20250110 20:57:12.433712 5594 raft_consensus.cc:357] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.434357 5594 raft_consensus.cc:383] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.434633 5594 raft_consensus.cc:738] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.444162 5595 ts_tablet_manager.cc:1918] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:12.444995 5595 log.cc:1198] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/2c1ca1670916401ab0a43685df1b6759
I20250110 20:57:12.445655 5597 ts_tablet_manager.cc:1397] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.338s user 0.007s sys 0.003s
I20250110 20:57:12.446309 5595 ts_tablet_manager.cc:1939] T 2c1ca1670916401ab0a43685df1b6759 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:12.448513 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 136) took 2170 ms. Trace:
I20250110 20:57:12.440651 5594 consensus_queue.cc:260] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.449807 5979 raft_consensus.cc:491] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.453809 5979 raft_consensus.cc:513] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.456966 5594 ts_tablet_manager.cc:1428] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.025s user 0.005s sys 0.000s
I20250110 20:57:12.458390 5594 tablet_bootstrap.cc:492] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.459462 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e361fb8b47d74548b5a2326d25a9e28c" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.460018 5172 raft_consensus.cc:2463] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.461014 5399 leader_election.cc:304] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:12.464313 5773 raft_consensus.cc:2267] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:12.469725 5979 leader_election.cc:290] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:12.471381 5977 raft_consensus.cc:513] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.471853 5145 tablet_service.cc:1467] Processing CreateTablet for tablet f80685cb3c494283bea421195bf7751f (DEFAULT_TABLE table=default.Table_a2199a41b9e74b16ab3d1b90b34ddad7 [id=75dcee4b73484274bac56a2ab1a827d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.472877 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80685cb3c494283bea421195bf7751f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.478303 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da24904dece04a6194790054f6d7153a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:12.478826 5172 raft_consensus.cc:3054] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:12.478791 5398 leader_election.cc:343] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: e361fb8b47d74548b5a2326d25a9e28c
W20250110 20:57:12.479827 5398 leader_election.cc:343] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: da24904dece04a6194790054f6d7153a
I20250110 20:57:12.473203 5977 leader_election.cc:290] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:12.480769 5980 raft_consensus.cc:2798] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.481158 5980 raft_consensus.cc:491] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.481009 5434 tablet_service.cc:1467] Processing CreateTablet for tablet 33db8a2fdd9c4fde985345604fd04215 (DEFAULT_TABLE table=default.Table_ba448520fdb24cd1be60910f20647848 [id=505b9ca1a68e401aa6f6f28a41f09181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.482290 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33db8a2fdd9c4fde985345604fd04215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.482537 5980 raft_consensus.cc:3054] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.485780 5172 raft_consensus.cc:2463] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:12.486744 5399 leader_election.cc:304] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:12.487782 5977 raft_consensus.cc:2798] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:12.488514 5977 raft_consensus.cc:695] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:12.490305 5977 consensus_queue.cc:237] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.494701 5594 tablet_bootstrap.cc:654] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.501739 5980 raft_consensus.cc:513] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.503788 5907 raft_consensus.cc:513] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:12.507625 5398 leader_election.cc:343] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:12.509792 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e361fb8b47d74548b5a2326d25a9e28c" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:12.510478 5174 raft_consensus.cc:3054] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:12.512967 5398 leader_election.cc:343] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: e361fb8b47d74548b5a2326d25a9e28c
I20250110 20:57:12.514184 5980 leader_election.cc:290] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:12.519465 5174 raft_consensus.cc:2463] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:12.509938 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a520230d9fd4a74914e46f784dcf747" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:12.520470 5399 leader_election.cc:304] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:12.520685 5172 raft_consensus.cc:3054] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.521391 5980 raft_consensus.cc:2798] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:12.522518 5980 raft_consensus.cc:695] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:12.524060 5980 consensus_queue.cc:237] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.529824 5773 ts_tablet_manager.cc:1905] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:12.539007 5594 tablet_bootstrap.cc:492] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.539412 5594 ts_tablet_manager.cc:1397] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.005s
I20250110 20:57:12.541791 5594 raft_consensus.cc:357] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.542505 5594 raft_consensus.cc:383] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.542224 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_0c447ed02fd84f509e75155308d4a9c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:12.542744 5594 raft_consensus.cc:738] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.543278 5594 consensus_queue.cc:260] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:12.543675 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c447ed02fd84f509e75155308d4a9c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:12.547258 4898 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_f1a0fe849b9c44a1b4834b9835cb49f9" } new_table_name: "default.Table_2fc3f8d84e404d6c85b99d9171a4d645" modify_external_catalogs: true
I20250110 20:57:12.550578 5594 ts_tablet_manager.cc:1428] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250110 20:57:12.550473 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e4c341fdb89a4e68bec331dde83cbfbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:12.552003 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4c341fdb89a4e68bec331dde83cbfbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:12.568434 5980 raft_consensus.cc:491] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.568859 5980 raft_consensus.cc:513] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.570361 5594 tablet_bootstrap.cc:492] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.458739 5597 raft_consensus.cc:357] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.573534 5980 leader_election.cc:290] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:12.573953 5597 raft_consensus.cc:383] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.574277 5597 raft_consensus.cc:738] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
W20250110 20:57:12.576078 5398 leader_election.cc:343] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 2967c06715e241dcbbe784eed2373704
I20250110 20:57:12.577262 5980 raft_consensus.cc:491] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.577247 5434 tablet_service.cc:1467] Processing CreateTablet for tablet f80685cb3c494283bea421195bf7751f (DEFAULT_TABLE table=default.Table_a2199a41b9e74b16ab3d1b90b34ddad7 [id=75dcee4b73484274bac56a2ab1a827d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.577719 5980 raft_consensus.cc:513] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.578384 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80685cb3c494283bea421195bf7751f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.581662 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:12.582978 5399 leader_election.cc:343] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.583489 5980 leader_election.cc:290] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.584115 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2967c06715e241dcbbe784eed2373704" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.584702 5174 raft_consensus.cc:2463] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.585620 5399 leader_election.cc:304] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:12.586454 5980 raft_consensus.cc:2798] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.586952 5980 raft_consensus.cc:491] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.587291 5980 raft_consensus.cc:3054] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.593083 5980 raft_consensus.cc:513] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:12.597270 5398 leader_election.cc:343] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 2967c06715e241dcbbe784eed2373704
I20250110 20:57:12.597991 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2967c06715e241dcbbe784eed2373704" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:12.599223 5174 raft_consensus.cc:3054] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.605257 5980 leader_election.cc:290] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:12.612008 5594 tablet_bootstrap.cc:654] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.615300 5174 raft_consensus.cc:2463] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:12.618189 5597 consensus_queue.cc:260] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:12.628201 5398 consensus_peers.cc:487] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31b65488c884452d876ced1c3d1efb9a. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:12.630103 5592 ts_tablet_manager.cc:1918] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:12.630463 5592 log.cc:1198] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:12.631085 5592 ts_tablet_manager.cc:1939] T 1ac07ac6fc9f477ba7f9accc5533be39 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:12.475759 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 33db8a2fdd9c4fde985345604fd04215 (DEFAULT_TABLE table=default.Table_ba448520fdb24cd1be60910f20647848 [id=505b9ca1a68e401aa6f6f28a41f09181]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.637705 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33db8a2fdd9c4fde985345604fd04215. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.637156 5595 rpcz_store.cc:276] 0110 20:57:10.277955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.285646 (+ 7691us) service_pool.cc:225] Handling call
0110 20:57:12.370589 (+2084943us) ts_tablet_manager.cc:1145] Deleting tablet 2c1ca1670916401ab0a43685df1b6759
0110 20:57:12.370634 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.425864 (+ 55230us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.440428 (+ 14564us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.448460 (+ 8032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084355}
I20250110 20:57:12.641788 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 2c1ca1670916401ab0a43685df1b6759 (table default.Table_792d0af8e9d04ad28d950c7656211372 [id=176f78029038442b932e8c92372948ef]) successfully deleted
I20250110 20:57:12.643878 5399 leader_election.cc:304] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:12.644760 5980 raft_consensus.cc:2798] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:12.645162 5980 raft_consensus.cc:695] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:12.645869 5980 consensus_queue.cc:237] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.655683 5594 tablet_bootstrap.cc:492] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.656064 5594 ts_tablet_manager.cc:1397] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.086s user 0.009s sys 0.000s
I20250110 20:57:12.347080 5929 raft_consensus.cc:513] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.515411 5880 raft_consensus.cc:491] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.580040 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:12.621735 5301 tablet_service.cc:1467] Processing CreateTablet for tablet f80685cb3c494283bea421195bf7751f (DEFAULT_TABLE table=default.Table_a2199a41b9e74b16ab3d1b90b34ddad7 [id=75dcee4b73484274bac56a2ab1a827d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:12.657650 5323 raft_consensus.cc:2463] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.658120 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f80685cb3c494283bea421195bf7751f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:12.658723 5398 leader_election.cc:304] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:12.659646 5977 raft_consensus.cc:2798] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.659946 5977 raft_consensus.cc:491] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.660219 5977 raft_consensus.cc:3054] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.657250 5880 raft_consensus.cc:513] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.666508 5594 raft_consensus.cc:357] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.667163 5594 raft_consensus.cc:383] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.667205 5880 leader_election.cc:290] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:12.667397 5594 raft_consensus.cc:738] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.667924 5594 consensus_queue.cc:260] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.668643 5172 raft_consensus.cc:2463] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:12.669036 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491626848ad54be3a18186c65d6dbb60" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:12.670225 5257 leader_election.cc:343] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.670945 5399 leader_election.cc:304] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:12.672196 5977 raft_consensus.cc:513] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.674582 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:12.675088 5323 raft_consensus.cc:3054] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.676362 5979 raft_consensus.cc:2798] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:12.678323 5597 ts_tablet_manager.cc:1428] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.232s user 0.006s sys 0.000s
I20250110 20:57:12.678380 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491626848ad54be3a18186c65d6dbb60" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:12.679136 5597 tablet_bootstrap.cc:492] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:12.681501 5255 leader_election.cc:343] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.681885 5255 leader_election.cc:304] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:12.681972 5979 raft_consensus.cc:695] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:12.682483 5595 tablet_replica.cc:331] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:12.683007 5880 raft_consensus.cc:2743] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:12.682740 5979 consensus_queue.cc:237] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.683184 5595 raft_consensus.cc:2238] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:12.683667 5595 raft_consensus.cc:2267] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:12.684285 5323 raft_consensus.cc:2463] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:12.685238 5398 leader_election.cc:304] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:12.686726 5989 raft_consensus.cc:2798] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:12.688527 5594 ts_tablet_manager.cc:1428] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250110 20:57:12.693045 5594 tablet_bootstrap.cc:492] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.694846 5977 leader_election.cc:290] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.695062 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:12.695428 5977 raft_consensus.cc:491] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.695544 5459 raft_consensus.cc:3054] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.695856 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:12.695784 5977 raft_consensus.cc:513] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.698329 5907 leader_election.cc:290] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:12.699137 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310cf52aa8c4930a7821117a46dc958" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.699702 5174 raft_consensus.cc:2463] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.700747 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310cf52aa8c4930a7821117a46dc958" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:12.702072 5459 raft_consensus.cc:2463] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
W20250110 20:57:12.702946 5399 leader_election.cc:343] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.703438 5323 raft_consensus.cc:2463] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.705224 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:12.706292 5257 leader_election.cc:343] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.708287 5929 leader_election.cc:290] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.707741 5398 leader_election.cc:304] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:12.709849 5977 leader_election.cc:290] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.722131 5977 raft_consensus.cc:2798] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.722498 5977 raft_consensus.cc:491] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.723620 5977 raft_consensus.cc:3054] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.710883 5989 raft_consensus.cc:695] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:12.728101 5989 consensus_queue.cc:237] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.730530 5255 leader_election.cc:304] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:12.731600 5929 raft_consensus.cc:2798] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:12.734481 4893 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:12.735124 5594 tablet_bootstrap.cc:654] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.735671 4893 catalog_manager.cc:6596] ReplaceTablet: tablet ae5956b1aa17449497f9695a111ebb66 deleted and replaced by tablet 4b28befa048244309a06f69b8a3b4b8f
I20250110 20:57:12.737497 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5956b1aa17449497f9695a111ebb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28befa048244309a06f69b8a3b4b8f) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:12.736816 5929 raft_consensus.cc:695] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Becoming Leader. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Running, Role: LEADER
I20250110 20:57:12.739809 4896 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:12.739017 5929 consensus_queue.cc:237] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.744752 4895 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:12.749892 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet a687769f69d844cc8564e61dc324d2b2
I20250110 20:57:12.751358 4897 master_service.cc:946] ReplaceTablet: received request to replace tablet f3104928188c489d93bfd499d148fa5d from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:12.753151 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet a687769f69d844cc8564e61dc324d2b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:12 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:12.753868 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5956b1aa17449497f9695a111ebb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28befa048244309a06f69b8a3b4b8f) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:12.754475 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet a687769f69d844cc8564e61dc324d2b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:12 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:12.755105 4895 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]). Expected version 2 got 0
I20250110 20:57:12.757560 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_1d59ab5eaa3845d4a405718a816829cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:12.758934 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d59ab5eaa3845d4a405718a816829cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:12.763687 5594 tablet_bootstrap.cc:492] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.764075 5594 ts_tablet_manager.cc:1397] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.004s
I20250110 20:57:12.764730 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e73d9dd6e1d44352936cc594ddd9de99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:12.766016 5939 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.052s sys 0.042s
W20250110 20:57:12.766076 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e73d9dd6e1d44352936cc594ddd9de99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:12.766306 5939 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s user 0.052s sys 0.042s
W20250110 20:57:12.769981 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33db8a2fdd9c4fde985345604fd04215 consensus state has no leader
I20250110 20:57:12.774315 5594 raft_consensus.cc:357] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.774978 5594 raft_consensus.cc:383] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.775226 5594 raft_consensus.cc:738] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.775897 4899 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 19d6eca349514b9fb0fff7cd6230027c
I20250110 20:57:12.775832 5594 consensus_queue.cc:260] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.777091 4899 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 187) took 1364 ms. Trace:
I20250110 20:57:12.778184 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6eca349514b9fb0fff7cd6230027c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:12.777369 4899 rpcz_store.cc:276] 0110 20:57:11.412535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.614783 (+202248us) service_pool.cc:225] Handling call
0110 20:57:11.660793 (+ 46010us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b616 00007f3bfd003b44 00007f3bfd003ad2 00007f3bfd003a01 00007f3bfd00397c 00007f3bfd002c32 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:11.696477 (+ 35684us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:11.742209 (+ 45732us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:11.742343 (+ 134us) catalog_manager.cc:2762] Locking tablets
0110 20:57:12.247505 (+505162us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:12.733863 (+486358us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:12.737224 (+ 3361us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:12.775823 (+ 38599us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:12.777042 (+ 1219us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:12.264379 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:12.264688 (+ 309us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:12.264709 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:57:12.264722 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:12.265010 (+ 288us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:12.265048 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:12.265070 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:12.265322 (+ 252us) tablet.cc:698] Row locks acquired
0110 20:57:12.265335 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:12.265459 (+ 124us) write_op.cc:270] Start()
0110 20:57:12.265538 (+ 79us) write_op.cc:276] Timestamp: P: 1736542632265388 usec, L: 0
0110 20:57:12.265552 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:12.537549 (+271997us) log.cc:844] Serialized 731 byte log entry
0110 20:57:12.546488 (+ 8939us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd4796d1 00007f3bfd4f0198 00007f3bfd95c3b1 00007f3bfd95bb88 00007f3bfd960821 00007f3bfd9607d9 00007f3bfd960769 00007f3bfd960731 00007f3bfd95f95d 000000000059ea04 000000000059e839
0110 20:57:12.727271 (+180783us) op_driver.cc:464] REPLICATION: finished
0110 20:57:12.728272 (+ 1001us) write_op.cc:301] APPLY: starting
0110 20:57:12.728326 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:12.728422 (+ 96us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:12.728440 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:12.728668 (+ 228us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:12.728802 (+ 134us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:12.728840 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:57:12.729027 (+ 187us) log.cc:844] Serialized 29 byte log entry
0110 20:57:12.729178 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:12.729238 (+ 60us) write_op.cc:454] Released schema lock
0110 20:57:12.729267 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:12.729498 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:12.729521 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":80077184,"child_traces":[["op",{"apply.queue_time_us":842,"apply.run_cpu_time_us":1188,"apply.run_wall_time_us":1187,"num_ops":2,"prepare.queue_time_us":15273,"prepare.run_cpu_time_us":4497,"prepare.run_wall_time_us":294793,"raft.queue_time_us":12449,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":8,"replication_time_us":461628,"spinlock_wait_cycles":624293760}]]}
I20250110 20:57:12.782260 5880 raft_consensus.cc:491] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.782660 5880 raft_consensus.cc:513] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.783742 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6eca349514b9fb0fff7cd6230027c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:12.784869 5880 leader_election.cc:290] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:12.786186 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:12.787425 5257 leader_election.cc:343] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.789088 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:12.791378 5977 raft_consensus.cc:513] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.793797 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310cf52aa8c4930a7821117a46dc958" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:12.794353 5319 raft_consensus.cc:3054] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.798602 5594 ts_tablet_manager.cc:1428] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.034s user 0.001s sys 0.005s
I20250110 20:57:12.802270 5594 tablet_bootstrap.cc:492] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.802517 5319 raft_consensus.cc:2463] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:12.803963 5398 leader_election.cc:304] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:12.805999 5989 raft_consensus.cc:2798] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:12.806416 5989 raft_consensus.cc:695] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:12.807108 5989 consensus_queue.cc:237] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.811761 5600 tablet_bootstrap.cc:492] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:12.812180 5600 ts_tablet_manager.cc:1397] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.515s user 0.072s sys 0.100s
W20250110 20:57:12.812117 5255 leader_election.cc:343] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:12.812556 5255 leader_election.cc:304] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:12.813391 5880 raft_consensus.cc:2743] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:12.819626 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6eca349514b9fb0fff7cd6230027c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:12.820626 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet a687769f69d844cc8564e61dc324d2b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:12 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:12.822031 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5956b1aa17449497f9695a111ebb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28befa048244309a06f69b8a3b4b8f) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:12.829388 5980 raft_consensus.cc:491] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.829870 5980 raft_consensus.cc:513] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.831751 5980 leader_election.cc:290] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.685322 5595 ts_tablet_manager.cc:1905] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:12.833128 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.834362 5594 tablet_bootstrap.cc:654] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.835079 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:12.835630 5319 raft_consensus.cc:2463] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:12.836548 5398 leader_election.cc:304] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
W20250110 20:57:12.838546 5399 leader_election.cc:343] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:12.840587 5110 consensus_peers.cc:487] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ac07ac6fc9f477ba7f9accc5533be39. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:12.841015 5989 raft_consensus.cc:2798] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:12.841339 5989 raft_consensus.cc:491] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:12.841295 5980 consensus_queue.cc:1035] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:57:12.841742 5989 raft_consensus.cc:3054] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.848343 5597 tablet_bootstrap.cc:654] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:12.851166 5977 leader_election.cc:290] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.854188 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 1ac07ac6fc9f477ba7f9accc5533be39 (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]) successfully deleted
I20250110 20:57:12.856395 5592 tablet_replica.cc:331] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:12.857127 5592 raft_consensus.cc:2238] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:12.857626 5592 raft_consensus.cc:2267] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:12.859143 5592 ts_tablet_manager.cc:1905] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:12.867766 5600 raft_consensus.cc:357] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.868515 5600 raft_consensus.cc:383] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.868805 5600 raft_consensus.cc:738] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.869474 5600 consensus_queue.cc:260] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:12.881085 4898 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7309d13f62524b6ab8b4c551728ecb6d consensus state has no leader
W20250110 20:57:12.907328 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_2fc3f8d84e404d6c85b99d9171a4d645 [id=1c2b9e8ff5d341c380b50023c577d0aa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7309d13f62524b6ab8b4c551728ecb6d consensus state has no leader
I20250110 20:57:12.918279 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet f3104928188c489d93bfd499d148fa5d
I20250110 20:57:12.919487 4897 catalog_manager.cc:6596] ReplaceTablet: tablet f3104928188c489d93bfd499d148fa5d deleted and replaced by tablet 29bbd640cb8e4adba77079181e8d1871
I20250110 20:57:12.920907 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet f3104928188c489d93bfd499d148fa5d with delete_type TABLET_DATA_DELETED (replaced by tablet 29bbd640cb8e4adba77079181e8d1871) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:12.926740 5595 ts_tablet_manager.cc:1918] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:12.927094 5595 log.cc:1198] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/cd0a666892d04e97a4b9967e38ab0946
I20250110 20:57:12.927687 5595 ts_tablet_manager.cc:1939] T cd0a666892d04e97a4b9967e38ab0946 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:12.930929 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 139) took 2633 ms. Trace:
I20250110 20:57:12.931209 5595 rpcz_store.cc:276] 0110 20:57:10.297902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.298005 (+ 103us) service_pool.cc:225] Handling call
0110 20:57:12.682350 (+2384345us) ts_tablet_manager.cc:1145] Deleting tablet cd0a666892d04e97a4b9967e38ab0946
0110 20:57:12.682391 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.901827 (+219436us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.926540 (+ 24713us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:12.930842 (+ 4302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2383838}
I20250110 20:57:12.932288 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet cd0a666892d04e97a4b9967e38ab0946 (table default.Table_c3b44e7a68ac47a1ad0f7c5496130418 [id=143a11fac7564c92b6a7224e6ac31bfb]) successfully deleted
I20250110 20:57:12.935963 5929 raft_consensus.cc:491] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.936404 5929 raft_consensus.cc:513] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:12.938176 5929 leader_election.cc:290] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:12.949357 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 140) took 2642 ms. Trace:
I20250110 20:57:12.949678 5595 rpcz_store.cc:276] 0110 20:57:10.307191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.307300 (+ 109us) service_pool.cc:225] Handling call
0110 20:57:12.949128 (+2641828us) ts_tablet_manager.cc:1145] Deleting tablet 8d67e94c8f9a414499b5e20a5cc0ead3
0110 20:57:12.949167 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.949309 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2641348}
I20250110 20:57:12.950780 5595 tablet_replica.cc:331] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
W20250110 20:57:12.951086 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 because the tablet was not found. No further retry: Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:12.961813 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d310cf52aa8c4930a7821117a46dc958" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:12.962370 5173 raft_consensus.cc:3054] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.987504 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637a39e4264b40dd9480dc13e37359d9" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:12.866271 5989 raft_consensus.cc:513] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.867126 5594 tablet_bootstrap.cc:492] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:12.989475 5594 ts_tablet_manager.cc:1397] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.187s user 0.010s sys 0.000s
I20250110 20:57:12.990528 5989 leader_election.cc:290] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:12.991703 5594 raft_consensus.cc:357] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.992369 5594 raft_consensus.cc:383] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:12.992614 5594 raft_consensus.cc:738] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:12.993163 5594 consensus_queue.cc:260] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:12.994724 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:12.995178 5594 ts_tablet_manager.cc:1428] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250110 20:57:12.995273 5323 raft_consensus.cc:3054] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:12.996321 5594 tablet_bootstrap.cc:492] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:12.997440 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet f3104928188c489d93bfd499d148fa5d with delete_type TABLET_DATA_DELETED (replaced by tablet 29bbd640cb8e4adba77079181e8d1871) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:12.921737 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet f3104928188c489d93bfd499d148fa5d with delete_type TABLET_DATA_DELETED (replaced by tablet 29bbd640cb8e4adba77079181e8d1871) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:12.928838 5977 raft_consensus.cc:491] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:12.999513 5977 raft_consensus.cc:513] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:12.939702 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637a39e4264b40dd9480dc13e37359d9" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:13.001650 5255 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.001822 5977 leader_election.cc:290] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:13.003763 5398 leader_election.cc:343] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:13.010121 5594 tablet_bootstrap.cc:654] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.013273 5929 raft_consensus.cc:491] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.013720 5929 raft_consensus.cc:513] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.016124 5929 leader_election.cc:290] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:13.018203 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:13.019413 5255 leader_election.cc:343] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.024981 5323 raft_consensus.cc:2463] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:13.026178 5398 leader_election.cc:304] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:13.027109 5989 raft_consensus.cc:2798] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:13.027559 5989 raft_consensus.cc:695] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:13.028290 5989 consensus_queue.cc:237] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.031025 5594 tablet_bootstrap.cc:492] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:13.031389 5594 ts_tablet_manager.cc:1397] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.004s
I20250110 20:57:13.034440 5594 raft_consensus.cc:357] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.035048 5594 raft_consensus.cc:383] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.035285 5594 raft_consensus.cc:738] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.040639 5594 consensus_queue.cc:260] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:13.005440 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 44928.2us
I20250110 20:57:13.049160 4891 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ead31c28470a4d3384b767df0ecce1ef" } modify_external_catalogs: true
I20250110 20:57:13.049695 4891 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ead31c28470a4d3384b767df0ecce1ef" } modify_external_catalogs: true
I20250110 20:57:13.050400 6014 tablet.cc:1721] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250110 20:57:13.035594 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 74292.5us
W20250110 20:57:13.052091 5257 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.052448 5257 leader_election.cc:304] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.052438 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_461c0135de1f4d318a4f6ceaf204ae45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:13.052811 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:13.053309 5929 raft_consensus.cc:2743] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:13.053822 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_461c0135de1f4d318a4f6ceaf204ae45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:13.054033 5257 leader_election.cc:343] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.054378 5257 leader_election.cc:304] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.055263 5929 raft_consensus.cc:2743] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:13.055752 5173 raft_consensus.cc:2463] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:13.056646 5977 raft_consensus.cc:491] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.057168 4892 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_338aa5da81e146be96e0f8cc5dcae996" } modify_external_catalogs: true
I20250110 20:57:13.057073 5977 raft_consensus.cc:513] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.057626 4892 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_338aa5da81e146be96e0f8cc5dcae996" } modify_external_catalogs: true
I20250110 20:57:13.058041 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5ba9f38c1493bba01180744f080e1" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:13.058712 5977 leader_election.cc:290] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:13.062681 5398 leader_election.cc:343] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:13.061693 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:13.064860 5399 leader_election.cc:343] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:13.065201 5111 consensus_peers.cc:487] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1ac07ac6fc9f477ba7f9accc5533be39. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:13.065213 5399 leader_election.cc:304] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:13.066191 5399 leader_election.cc:343] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.066769 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d67e94c8f9a414499b5e20a5cc0ead3" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:13.066872 5977 raft_consensus.cc:2743] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:13.070272 5399 leader_election.cc:343] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.070618 5399 leader_election.cc:304] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.071192 5977 raft_consensus.cc:2743] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:13.071180 4897 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ce0a88d3ee2c4c17996b244999136619" } new_table_name: "default.Table_6fc93557ff4c413284a1e596711e5753" modify_external_catalogs: true
I20250110 20:57:13.072738 5594 ts_tablet_manager.cc:1428] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.041s user 0.006s sys 0.001s
I20250110 20:57:13.074621 5594 tablet_bootstrap.cc:492] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:13.075294 6018 tablet.cc:1721] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:13.079643 5594 tablet_bootstrap.cc:654] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.085608 4899 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a2199a41b9e74b16ab3d1b90b34ddad7" } new_table_name: "default.Table_3f641bf1cdfd46fda1bd487fa4f01f1e" modify_external_catalogs: true
I20250110 20:57:13.117691 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_68999c6dd9a84b24b631302d58de6c02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:13.119088 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68999c6dd9a84b24b631302d58de6c02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:13.123019 5594 tablet_bootstrap.cc:492] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:13.123383 5594 ts_tablet_manager.cc:1397] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.002s
I20250110 20:57:13.125483 5594 raft_consensus.cc:357] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.126125 5594 raft_consensus.cc:383] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.126360 5594 raft_consensus.cc:738] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.126888 5594 consensus_queue.cc:260] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.127684 5977 raft_consensus.cc:491] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.128065 5977 raft_consensus.cc:513] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.129779 5977 leader_election.cc:290] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.132328 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cbb46ea11c4e7d8bc5993ccdc52247" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:13.132920 5319 raft_consensus.cc:2463] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:13.134382 5398 leader_election.cc:304] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:13.135181 5977 raft_consensus.cc:2798] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:13.135465 5977 raft_consensus.cc:491] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:13.135735 5977 raft_consensus.cc:3054] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:13.138438 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cbb46ea11c4e7d8bc5993ccdc52247" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:13.140309 5399 leader_election.cc:343] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.143009 5977 raft_consensus.cc:513] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.152148 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cbb46ea11c4e7d8bc5993ccdc52247" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:13.153170 5399 leader_election.cc:343] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.150285 5594 ts_tablet_manager.cc:1428] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.027s user 0.006s sys 0.001s
I20250110 20:57:13.154109 6022 tablet.cc:1721] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:13.157259 5594 tablet_bootstrap.cc:492] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:13.156441 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cbb46ea11c4e7d8bc5993ccdc52247" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:13.158175 5319 raft_consensus.cc:3054] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:13.161480 5977 leader_election.cc:290] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.164129 5319 raft_consensus.cc:2463] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:13.164983 5595 raft_consensus.cc:2238] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:13.165112 5398 leader_election.cc:304] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.165442 5595 raft_consensus.cc:2267] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:13.166179 5977 raft_consensus.cc:2798] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:13.166921 5595 ts_tablet_manager.cc:1905] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:13.166759 5977 raft_consensus.cc:695] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:13.167517 5929 raft_consensus.cc:491] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.167688 5977 consensus_queue.cc:237] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.167914 5929 raft_consensus.cc:513] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.169554 5929 leader_election.cc:290] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:13.170253 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b954dbccb45b2a6ab7cc5cceadffb" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:13.171554 5257 leader_election.cc:343] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.172152 5594 tablet_bootstrap.cc:654] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.179656 5600 ts_tablet_manager.cc:1428] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.367s user 0.006s sys 0.000s
I20250110 20:57:13.181389 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b954dbccb45b2a6ab7cc5cceadffb" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:13.186898 5255 leader_election.cc:343] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.187295 5255 leader_election.cc:304] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.188215 5929 raft_consensus.cc:2743] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:13.190510 5600 tablet_bootstrap.cc:492] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:13.199045 5597 tablet_bootstrap.cc:492] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:13.199429 5597 ts_tablet_manager.cc:1397] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.520s user 0.008s sys 0.000s
I20250110 20:57:13.199736 5594 tablet_bootstrap.cc:492] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:13.200109 5594 ts_tablet_manager.cc:1397] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.001s
I20250110 20:57:13.201684 5597 raft_consensus.cc:357] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.202316 5597 raft_consensus.cc:383] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.202188 5594 raft_consensus.cc:357] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.202549 5597 raft_consensus.cc:738] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.202788 5594 raft_consensus.cc:383] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.203019 5594 raft_consensus.cc:738] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.203171 5597 consensus_queue.cc:260] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.203547 5594 consensus_queue.cc:260] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.206341 5597 ts_tablet_manager.cc:1428] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20250110 20:57:13.207062 5597 tablet_bootstrap.cc:492] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:13.205365 5594 ts_tablet_manager.cc:1428] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250110 20:57:13.210451 5594 tablet_bootstrap.cc:492] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:13.213076 5597 tablet_bootstrap.cc:654] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.220642 5773 ts_tablet_manager.cc:1918] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:13.227814 5594 tablet_bootstrap.cc:654] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.252530 5594 tablet_bootstrap.cc:492] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:13.252903 5594 ts_tablet_manager.cc:1397] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.044s user 0.008s sys 0.003s
I20250110 20:57:13.255015 5594 raft_consensus.cc:357] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.255753 5594 raft_consensus.cc:383] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.256004 5594 raft_consensus.cc:738] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.256538 5594 consensus_queue.cc:260] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.258733 5594 ts_tablet_manager.cc:1428] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250110 20:57:13.265267 5990 raft_consensus.cc:491] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.264653 4898 catalog_manager.cc:5526] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:13.265959 4898 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]). Expected version 1 got 0
I20250110 20:57:13.265733 5990 raft_consensus.cc:513] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.266592 4898 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:13.267493 5990 leader_election.cc:290] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
W20250110 20:57:13.270277 5398 leader_election.cc:343] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:13.264036 5597 tablet_bootstrap.cc:492] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:13.271102 5773 log.cc:1198] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/d9776df98cb54be38a521852634490e7
I20250110 20:57:13.268692 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf0c87159574bfcbc7a34abe508f9d6" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:13.271910 5773 ts_tablet_manager.cc:1939] T d9776df98cb54be38a521852634490e7 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
W20250110 20:57:13.272244 5399 leader_election.cc:343] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.272588 5399 leader_election.cc:304] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.272732 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 151) took 1668 ms. Trace:
I20250110 20:57:13.271209 5597 ts_tablet_manager.cc:1397] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.064s user 0.026s sys 0.023s
I20250110 20:57:13.273559 5990 raft_consensus.cc:2743] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:13.273206 5773 rpcz_store.cc:276] 0110 20:57:11.604186 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.604285 (+ 99us) service_pool.cc:225] Handling call
0110 20:57:12.218858 (+614573us) ts_tablet_manager.cc:1145] Deleting tablet d9776df98cb54be38a521852634490e7
0110 20:57:12.218894 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:12.713745 (+494851us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.220438 (+506693us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.272681 (+ 52243us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":614097}
I20250110 20:57:13.274813 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 152) took 1669 ms. Trace:
I20250110 20:57:13.275316 5597 raft_consensus.cc:357] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.276412 5594 tablet_bootstrap.cc:492] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:13.275930 5597 raft_consensus.cc:383] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.277309 5597 raft_consensus.cc:738] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.277827 5597 consensus_queue.cc:260] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.280138 5597 ts_tablet_manager.cc:1428] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20250110 20:57:13.280891 5597 tablet_bootstrap.cc:492] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:13.281365 5594 tablet_bootstrap.cc:654] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.288880 5597 tablet_bootstrap.cc:654] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.319439 5597 tablet_bootstrap.cc:492] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:13.319797 5597 ts_tablet_manager.cc:1397] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.006s
I20250110 20:57:13.321789 5597 raft_consensus.cc:357] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.322398 5597 raft_consensus.cc:383] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.322723 5597 raft_consensus.cc:738] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.323329 5597 consensus_queue.cc:260] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.325651 5597 ts_tablet_manager.cc:1428] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20250110 20:57:13.329111 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet d9776df98cb54be38a521852634490e7 (table default.Table_ffc1b6cd4c7f4a7a9ce00d4d5bc0076f [id=3bbd1acd709b44899907148df944b20d]) successfully deleted
I20250110 20:57:13.329741 5597 tablet_bootstrap.cc:492] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:13.332335 5595 ts_tablet_manager.cc:1918] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:13.332751 5595 log.cc:1198] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/2d634add82994f74960cd2b8373a4dc6
I20250110 20:57:13.333336 5595 ts_tablet_manager.cc:1939] T 2d634add82994f74960cd2b8373a4dc6 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:13.334440 5597 tablet_bootstrap.cc:654] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.267241 4898 catalog_manager.cc:5526] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:13.348474 4898 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]). Expected version 1 got 0
I20250110 20:57:13.349114 4898 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:13.275084 5773 rpcz_store.cc:276] 0110 20:57:11.605022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.605115 (+ 93us) service_pool.cc:225] Handling call
0110 20:57:13.274223 (+1669108us) ts_tablet_manager.cc:1145] Deleting tablet 78538b2e7bcc45c7afaa145bad6e5813
0110 20:57:13.274259 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.274765 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668620}
I20250110 20:57:13.356914 5773 tablet_replica.cc:331] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:13.357647 5773 raft_consensus.cc:2238] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:13.358068 5773 raft_consensus.cc:2267] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:13.359650 5773 ts_tablet_manager.cc:1905] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:13.364780 5597 tablet_bootstrap.cc:492] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:13.365142 5597 ts_tablet_manager.cc:1397] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.003s
I20250110 20:57:13.371685 4892 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 29bbd640cb8e4adba77079181e8d1871
W20250110 20:57:13.377146 4899 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f80685cb3c494283bea421195bf7751f consensus state has no leader
W20250110 20:57:13.392124 4897 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fdb2ee9419244a9ab939be56b2d7c72 has no consensus state
I20250110 20:57:13.393108 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a0d2f2c0efd3421980eae4849cea168a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:13.394552 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0d2f2c0efd3421980eae4849cea168a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:13.403187 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 78538b2e7bcc45c7afaa145bad6e5813 because the tablet was not found. No further retry: Not found: Tablet not found: 78538b2e7bcc45c7afaa145bad6e5813
I20250110 20:57:13.405781 4898 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:13.423239 5600 tablet_bootstrap.cc:654] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.446143 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 138) took 3157 ms. Trace:
I20250110 20:57:13.446496 5595 rpcz_store.cc:276] 0110 20:57:10.288953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.311748 (+ 22795us) service_pool.cc:225] Handling call
0110 20:57:12.950655 (+2638907us) ts_tablet_manager.cc:1145] Deleting tablet 2d634add82994f74960cd2b8373a4dc6
0110 20:57:12.950688 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.176550 (+225862us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.332106 (+155556us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.446081 (+113975us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":324452,"tablet-delete.queue_time_us":2638396}
I20250110 20:57:13.447880 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 2d634add82994f74960cd2b8373a4dc6 (table default.Table_094c089272f6475a989a44b5e1843fd5 [id=505b9ca1a68e401aa6f6f28a41f09181]) successfully deleted
I20250110 20:57:13.448999 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 150) took 2169 ms. Trace:
I20250110 20:57:13.449375 5595 rpcz_store.cc:276] 0110 20:57:11.279855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.279961 (+ 106us) service_pool.cc:225] Handling call
0110 20:57:13.448684 (+2168723us) ts_tablet_manager.cc:1145] Deleting tablet c813130649cc4df1b483ac4a86b4f0a0
0110 20:57:13.448725 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.448950 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2168252}
W20250110 20:57:13.450999 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet c813130649cc4df1b483ac4a86b4f0a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c813130649cc4df1b483ac4a86b4f0a0 already in progress: creating tablet
I20250110 20:57:13.451041 5581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:13.451821 5595 tablet_replica.cc:331] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:13.453339 5581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:13.453907 5581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:13.463099 5594 tablet_bootstrap.cc:492] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:13.463583 5594 ts_tablet_manager.cc:1397] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.187s user 0.017s sys 0.009s
I20250110 20:57:13.466195 5594 raft_consensus.cc:357] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.466900 5594 raft_consensus.cc:383] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.467211 5594 raft_consensus.cc:738] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.467800 5594 consensus_queue.cc:260] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.472559 5990 consensus_queue.cc:1035] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:13.489881 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d08622be3a1c4350b1df15c8f949733d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:13.491393 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d08622be3a1c4350b1df15c8f949733d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:13.512529 5595 raft_consensus.cc:2238] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:13.516412 5597 raft_consensus.cc:357] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.517048 5597 raft_consensus.cc:383] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.517297 5597 raft_consensus.cc:738] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.541195 5594 ts_tablet_manager.cc:1428] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.077s user 0.005s sys 0.000s
I20250110 20:57:13.542259 5594 tablet_bootstrap.cc:492] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:13.549705 5595 raft_consensus.cc:2267] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:13.550601 5597 consensus_queue.cc:260] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.511528 5600 tablet_bootstrap.cc:492] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:13.551571 5600 ts_tablet_manager.cc:1397] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.361s user 0.009s sys 0.000s
I20250110 20:57:13.554421 5600 raft_consensus.cc:357] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.555226 5600 raft_consensus.cc:383] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.555514 5600 raft_consensus.cc:738] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.556229 5600 consensus_queue.cc:260] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.559128 5600 ts_tablet_manager.cc:1428] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:57:13.561625 5597 ts_tablet_manager.cc:1428] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.196s user 0.025s sys 0.038s
I20250110 20:57:13.564435 5597 tablet_bootstrap.cc:492] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:13.571348 4900 master_service.cc:946] ReplaceTablet: received request to replace tablet d310cf52aa8c4930a7821117a46dc958 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:13.574012 5597 tablet_bootstrap.cc:654] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.587193 5597 tablet_bootstrap.cc:492] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:13.587538 5597 ts_tablet_manager.cc:1397] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20250110 20:57:13.589524 5597 raft_consensus.cc:357] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.590113 5597 raft_consensus.cc:383] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.590351 5597 raft_consensus.cc:738] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.591010 5597 consensus_queue.cc:260] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.598709 5597 ts_tablet_manager.cc:1428] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250110 20:57:13.599437 5597 tablet_bootstrap.cc:492] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:13.606748 5597 tablet_bootstrap.cc:654] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.461323 5592 ts_tablet_manager.cc:1918] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:13.610030 5592 log.cc:1198] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/2d634add82994f74960cd2b8373a4dc6
W20250110 20:57:13.610249 5398 consensus_peers.cc:487] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e361fb8b47d74548b5a2326d25a9e28c. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:13.624152 5592 ts_tablet_manager.cc:1939] T 2d634add82994f74960cd2b8373a4dc6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:13.625990 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 157) took 1656 ms. Trace:
I20250110 20:57:13.630630 5989 raft_consensus.cc:491] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.631068 5989 raft_consensus.cc:513] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.632802 5989 leader_election.cc:290] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.644821 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527bbddda8184a6eb67c2411ae50db63" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:13.646102 5398 consensus_peers.cc:487] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da24904dece04a6194790054f6d7153a. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:13.646265 5399 leader_election.cc:343] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.647670 5987 raft_consensus.cc:491] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.648078 5987 raft_consensus.cc:513] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.649798 5987 leader_election.cc:290] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:13.609462 5929 raft_consensus.cc:491] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.651595 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf0c87159574bfcbc7a34abe508f9d6" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:13.651574 5929 raft_consensus.cc:513] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.652130 5459 raft_consensus.cc:2463] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:13.653301 5929 leader_election.cc:290] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:13.653389 5110 leader_election.cc:304] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:13.623136 5595 ts_tablet_manager.cc:1905] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:13.634261 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527bbddda8184a6eb67c2411ae50db63" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
W20250110 20:57:13.655632 5398 leader_election.cc:343] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.655999 5398 leader_election.cc:304] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.658883 5987 raft_consensus.cc:2798] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:13.659200 5987 raft_consensus.cc:491] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:13.659497 5987 raft_consensus.cc:3054] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:13.660070 5990 raft_consensus.cc:2743] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:13.668354 4897 master_service.cc:946] ReplaceTablet: received request to replace tablet 48e53d95ef8645c0bd123138655305ac from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:13.672417 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde470279464496aa7efdee3800754f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:13.677683 5600 tablet_bootstrap.cc:492] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:13.682684 5990 consensus_queue.cc:1035] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:13.626287 5592 rpcz_store.cc:276] 0110 20:57:11.969544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.969658 (+ 114us) service_pool.cc:225] Handling call
0110 20:57:12.854880 (+885222us) ts_tablet_manager.cc:1145] Deleting tablet 2d634add82994f74960cd2b8373a4dc6
0110 20:57:12.856281 (+ 1401us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.339850 (+483569us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.461098 (+121248us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.625933 (+164835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":884730}
I20250110 20:57:13.695514 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 2d634add82994f74960cd2b8373a4dc6 (table default.Table_094c089272f6475a989a44b5e1843fd5 [id=505b9ca1a68e401aa6f6f28a41f09181]) successfully deleted
I20250110 20:57:13.696349 5592 tablet_replica.cc:331] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:13.697091 5592 raft_consensus.cc:2238] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:13.698020 5592 raft_consensus.cc:2267] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:13.700357 5592 ts_tablet_manager.cc:1905] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:13.717748 5597 tablet_bootstrap.cc:492] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:13.718130 5597 ts_tablet_manager.cc:1397] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.119s user 0.013s sys 0.021s
I20250110 20:57:13.720129 5597 raft_consensus.cc:357] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.720743 5597 raft_consensus.cc:383] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.720995 5597 raft_consensus.cc:738] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.723215 5597 consensus_queue.cc:260] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.728391 5773 ts_tablet_manager.cc:1918] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:13.781669 5773 log.cc:1198] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:13.734054 5595 ts_tablet_manager.cc:1918] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:13.782845 5595 log.cc:1198] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/527bbddda8184a6eb67c2411ae50db63
I20250110 20:57:13.783469 5595 ts_tablet_manager.cc:1939] T 527bbddda8184a6eb67c2411ae50db63 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:13.785138 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 151) took 2505 ms. Trace:
W20250110 20:57:13.704036 5398 consensus_peers.cc:487] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2967c06715e241dcbbe784eed2373704. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:13.786689 5398 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 82724.9us
I20250110 20:57:13.723455 5977 raft_consensus.cc:491] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.787945 5977 raft_consensus.cc:513] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.789620 5977 leader_election.cc:290] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.791158 4893 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c" } modify_external_catalogs: true
I20250110 20:57:13.791692 4893 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c" } modify_external_catalogs: true
W20250110 20:57:13.782405 5257 leader_election.cc:343] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:13.793555 5255 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 156192us
I20250110 20:57:13.785392 5595 rpcz_store.cc:276] 0110 20:57:11.280161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.280520 (+ 359us) service_pool.cc:225] Handling call
0110 20:57:13.451687 (+2171167us) ts_tablet_manager.cc:1145] Deleting tablet 527bbddda8184a6eb67c2411ae50db63
0110 20:57:13.451724 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.674436 (+222712us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.733850 (+ 59414us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.785085 (+ 51235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2170654}
I20250110 20:57:13.794759 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde470279464496aa7efdee3800754f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:13.795289 5459 raft_consensus.cc:2463] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:13.796358 5595 tablet_replica.cc:331] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:13.548368 5594 tablet_bootstrap.cc:654] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.769667 5979 raft_consensus.cc:491] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.797138 5979 raft_consensus.cc:513] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.799690 5979 leader_election.cc:290] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.799299 5595 raft_consensus.cc:2238] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:13.811808 5594 tablet_bootstrap.cc:492] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:13.812310 5594 ts_tablet_manager.cc:1397] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.270s user 0.001s sys 0.012s
I20250110 20:57:13.818858 5255 leader_election.cc:304] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:13.819267 5595 raft_consensus.cc:2267] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:13.819619 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 527bbddda8184a6eb67c2411ae50db63 (table default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]) successfully deleted
I20250110 20:57:13.820735 5595 ts_tablet_manager.cc:1905] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:13.828648 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet d310cf52aa8c4930a7821117a46dc958
I20250110 20:57:13.836174 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet d310cf52aa8c4930a7821117a46dc958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4aaa42bc5ab042e79baee1ea838851eb) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:13.833987 5594 raft_consensus.cc:357] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.838630 4900 catalog_manager.cc:6596] ReplaceTablet: tablet d310cf52aa8c4930a7821117a46dc958 deleted and replaced by tablet 4aaa42bc5ab042e79baee1ea838851eb
I20250110 20:57:13.840046 5600 tablet_bootstrap.cc:654] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.840319 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1ec05f4286b94b13bbe215ff5cba4d32" } new_table_name: "default.Table_a9b22870bca84943b105815613f88adb" modify_external_catalogs: true
I20250110 20:57:13.846190 5929 raft_consensus.cc:2798] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:13.846578 5929 raft_consensus.cc:491] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:13.846868 5929 raft_consensus.cc:3054] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:13.852720 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951d5ca47e8409db646b43bb04a7034" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:13.853977 5979 consensus_queue.cc:1035] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:13.855623 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:13.856165 5319 raft_consensus.cc:2463] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:13.856796 5320 raft_consensus.cc:2463] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:13.857064 5398 leader_election.cc:304] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:13.858155 5398 leader_election.cc:304] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:13.858840 5977 raft_consensus.cc:2798] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:13.859144 5977 raft_consensus.cc:491] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:13.859450 5977 raft_consensus.cc:3054] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:13.838095 5594 raft_consensus.cc:383] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.862411 5594 raft_consensus.cc:738] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.862994 5594 consensus_queue.cc:260] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.866195 5600 tablet_bootstrap.cc:492] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:13.866582 5600 ts_tablet_manager.cc:1397] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.189s user 0.007s sys 0.002s
I20250110 20:57:13.868680 5600 raft_consensus.cc:357] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.869306 5600 raft_consensus.cc:383] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.869565 5600 raft_consensus.cc:738] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.872438 5990 raft_consensus.cc:2798] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:13.872741 5990 raft_consensus.cc:491] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:13.873034 5990 raft_consensus.cc:3054] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:13.873063 5600 consensus_queue.cc:260] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.771219 5597 ts_tablet_manager.cc:1428] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.053s user 0.002s sys 0.003s
I20250110 20:57:13.875479 5597 tablet_bootstrap.cc:492] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:13.875384 5977 raft_consensus.cc:513] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.879832 5990 raft_consensus.cc:513] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.882462 5597 tablet_bootstrap.cc:654] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:13.884038 5990 leader_election.cc:290] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.771518 6054 tablet.cc:1721] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:13.680048 5987 raft_consensus.cc:513] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.873380 5929 raft_consensus.cc:513] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.890167 5987 leader_election.cc:290] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:13.890489 5773 ts_tablet_manager.cc:1939] T 3a520230d9fd4a74914e46f784dcf747 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:13.890616 5987 raft_consensus.cc:491] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.891247 5977 leader_election.cc:290] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.891057 5987 raft_consensus.cc:513] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.892253 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 153) took 2286 ms. Trace:
I20250110 20:57:13.892735 5987 leader_election.cc:290] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:13.791357 5111 leader_election.cc:343] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
W20250110 20:57:13.893760 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 102435us
I20250110 20:57:13.895483 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951d5ca47e8409db646b43bb04a7034" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:13.896016 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:13.896940 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951d5ca47e8409db646b43bb04a7034" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:13.791936 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:13.805279 6049 raft_consensus.cc:491] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.838124 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet d310cf52aa8c4930a7821117a46dc958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4aaa42bc5ab042e79baee1ea838851eb) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:13.898474 6049 raft_consensus.cc:513] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:13.900285 6049 leader_election.cc:290] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:13.902375 5929 leader_election.cc:290] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:13.903728 5399 leader_election.cc:343] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:13.904480 5399 leader_election.cc:343] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:13.905052 5399 consensus_peers.cc:487] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a520230d9fd4a74914e46f784dcf747. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:13.902812 5600 ts_tablet_manager.cc:1428] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
W20250110 20:57:13.905516 5399 leader_election.cc:343] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.906246 5600 tablet_bootstrap.cc:492] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:13.907708 5977 consensus_queue.cc:1035] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:57:13.908597 5111 leader_election.cc:343] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: f0b5ba9f38c1493bba01180744f080e1
W20250110 20:57:13.909170 5111 leader_election.cc:343] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
W20250110 20:57:13.910841 5399 leader_election.cc:343] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:13.911367 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb7f09df8e274f3e9542292d5c1ec6c6 consensus state has no leader
I20250110 20:57:13.913301 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e4c341fdb89a4e68bec331dde83cbfbc" } new_table_name: "default.Table_03ca54bede3b459497f750a83fe9e117" modify_external_catalogs: true
I20250110 20:57:13.774732 6055 tablet.cc:1721] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:13.775481 5989 consensus_queue.cc:1035] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:13.932898 6064 raft_consensus.cc:491] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:13.933379 6064 raft_consensus.cc:513] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:13.936321 5170 raft_consensus.cc:3054] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:13.939368 5597 tablet_bootstrap.cc:492] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:13.939750 5597 ts_tablet_manager.cc:1397] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.002s
I20250110 20:57:13.941924 5597 raft_consensus.cc:357] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:13.942576 5597 raft_consensus.cc:383] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:13.942842 5597 raft_consensus.cc:738] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:13.943332 5597 consensus_queue.cc:260] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:13.952466 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 137187us
I20250110 20:57:13.953714 5990 consensus_queue.cc:1035] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:13.963320 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1b57de9e4b4b2ab6c3bce8fe78166a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:13.964524 5399 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:13.971201 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cf0c87159574bfcbc7a34abe508f9d6" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:13.971755 5458 raft_consensus.cc:3054] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:13.973707 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_a9b22870bca84943b105815613f88adb [id=2c78ee52e44a46fe877f474a7ecb771b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet cb7f09df8e274f3e9542292d5c1ec6c6 consensus state has no leader
I20250110 20:57:13.978739 5458 raft_consensus.cc:2463] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:13.980486 6064 leader_election.cc:290] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:13.892529 5773 rpcz_store.cc:276] 0110 20:57:11.605849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.605937 (+ 88us) service_pool.cc:225] Handling call
0110 20:57:13.356771 (+1750834us) ts_tablet_manager.cc:1145] Deleting tablet 3a520230d9fd4a74914e46f784dcf747
0110 20:57:13.356810 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.647160 (+290350us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.728184 (+ 81024us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:13.892189 (+164005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750364}
W20250110 20:57:13.995112 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c1330e66fc14b4da9e4a4669943360c has no consensus state
I20250110 20:57:13.995013 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:13.997702 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_46ff9199b08c4e779e9e21081434e52e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:14.001042 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46ff9199b08c4e779e9e21081434e52e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:14.005714 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5ba9f38c1493bba01180744f080e1" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:14.006398 5459 raft_consensus.cc:2463] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:13.989956 6048 raft_consensus.cc:491] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.011865 6048 raft_consensus.cc:513] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:14.013798 6048 leader_election.cc:290] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:14.014269 5597 ts_tablet_manager.cc:1428] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.074s user 0.005s sys 0.001s
I20250110 20:57:14.015125 5597 tablet_bootstrap.cc:492] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:14.015693 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 154) took 2084 ms. Trace:
I20250110 20:57:14.015882 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 3a520230d9fd4a74914e46f784dcf747 (table default.Table_8df7bbf6722249b2be68514a607459b9 [id=64cfe285ff2041778f15cf7e6020e9cb]) successfully deleted
I20250110 20:57:14.016058 5773 rpcz_store.cc:276] 0110 20:57:11.931402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.932674 (+ 1272us) service_pool.cc:225] Handling call
0110 20:57:14.015385 (+2082711us) ts_tablet_manager.cc:1145] Deleting tablet 5c5b954dbccb45b2a6ab7cc5cceadffb
0110 20:57:14.015424 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.015644 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2081999}
W20250110 20:57:14.017130 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c5b954dbccb45b2a6ab7cc5cceadffb already in progress: creating tablet
I20250110 20:57:14.017822 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 155) took 2081 ms. Trace:
I20250110 20:57:14.018252 5773 rpcz_store.cc:276] 0110 20:57:11.936194 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.936319 (+ 125us) service_pool.cc:225] Handling call
0110 20:57:14.017219 (+2080900us) ts_tablet_manager.cc:1145] Deleting tablet c951d5ca47e8409db646b43bb04a7034
0110 20:57:14.017255 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.017776 (+ 521us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2080408}
I20250110 20:57:13.988152 5594 ts_tablet_manager.cc:1428] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.176s user 0.005s sys 0.000s
I20250110 20:57:14.020314 5594 tablet_bootstrap.cc:492] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:14.023111 5592 ts_tablet_manager.cc:1918] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:14.023883 5592 log.cc:1198] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:14.025394 5600 tablet_bootstrap.cc:654] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.025892 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:14.026371 5592 ts_tablet_manager.cc:1939] T 31b65488c884452d876ced1c3d1efb9a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:14.019371 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7462d21b91c2441fb9e11383466e9ca7" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:14.040625 5316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:14.024365 5170 raft_consensus.cc:3054] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.022817 5929 raft_consensus.cc:491] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.041173 5316 raft_consensus.cc:3054] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.026974 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde470279464496aa7efdee3800754f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:14.031912 5171 raft_consensus.cc:3054] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.058878 5310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:14.050783 5929 raft_consensus.cc:513] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.061269 5929 leader_election.cc:290] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:14.041780 5110 leader_election.cc:304] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:14.047570 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951d5ca47e8409db646b43bb04a7034" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:14.026696 5458 raft_consensus.cc:2388] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac9003be04f340f3886b5625cc928eb0 in current term 1: Already voted for candidate 46005a5920cc4a3697a5c61b1c271384 in this term.
I20250110 20:57:14.062369 5319 raft_consensus.cc:3054] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.051604 6064 consensus_queue.cc:1035] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:14.062665 6048 raft_consensus.cc:2798] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:14.063082 6048 raft_consensus.cc:695] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:14.063367 5989 consensus_queue.cc:1035] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:14.063763 6048 consensus_queue.cc:237] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:14.065274 5320 raft_consensus.cc:2388] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac9003be04f340f3886b5625cc928eb0 in current term 1: Already voted for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in this term.
I20250110 20:57:14.057773 5979 consensus_queue.cc:1035] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:14.069934 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 155) took 2102 ms. Trace:
I20250110 20:57:14.075281 5773 tablet_replica.cc:331] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:14.075084 6054 tablet.cc:1721] T 04fd9785888141abaab655f0ba74efb9 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:14.076009 5773 raft_consensus.cc:2238] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:14.076821 5773 raft_consensus.cc:2267] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.031666 5595 ts_tablet_manager.cc:1918] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:14.085569 5595 log.cc:1198] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/491626848ad54be3a18186c65d6dbb60
I20250110 20:57:14.086292 5595 ts_tablet_manager.cc:1939] T 491626848ad54be3a18186c65d6dbb60 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:14.086678 5316 raft_consensus.cc:2463] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
W20250110 20:57:14.072208 5398 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:14.089776 5398 leader_election.cc:304] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:14.091467 5398 consensus_peers.cc:487] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a520230d9fd4a74914e46f784dcf747. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:14.092203 6064 raft_consensus.cc:2743] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.092665 6064 raft_consensus.cc:491] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.093007 6064 raft_consensus.cc:513] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.094652 6064 leader_election.cc:290] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:14.095059 6064 consensus_queue.cc:1035] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:14.095758 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a666892d04e97a4b9967e38ab0946" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:14.096016 5773 ts_tablet_manager.cc:1905] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:14.096784 5399 leader_election.cc:343] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:14.101851 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet c951d5ca47e8409db646b43bb04a7034 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c951d5ca47e8409db646b43bb04a7034 already in progress: creating tablet
I20250110 20:57:14.077183 6048 raft_consensus.cc:491] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.104313 6048 raft_consensus.cc:513] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:14.105686 5255 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 286866us
I20250110 20:57:14.101384 5319 raft_consensus.cc:2463] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:14.106343 5398 leader_election.cc:304] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.107141 5989 raft_consensus.cc:2798] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:14.107542 5989 raft_consensus.cc:695] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:14.108582 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet d310cf52aa8c4930a7821117a46dc958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4aaa42bc5ab042e79baee1ea838851eb) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:14.107667 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde470279464496aa7efdee3800754f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:14.107841 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:14.109591 5459 raft_consensus.cc:3054] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:14.110188 5255 leader_election.cc:343] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.115357 5600 tablet_bootstrap.cc:492] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:14.115716 5600 ts_tablet_manager.cc:1397] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.210s user 0.010s sys 0.000s
I20250110 20:57:14.118438 6048 leader_election.cc:290] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:14.118629 5600 raft_consensus.cc:357] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.119231 5600 raft_consensus.cc:383] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.119462 5600 raft_consensus.cc:738] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.119421 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:14.120138 5600 consensus_queue.cc:260] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.082811 5110 leader_election.cc:304] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:14.108287 5989 consensus_queue.cc:237] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.123322 5600 ts_tablet_manager.cc:1428] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250110 20:57:14.127594 5398 leader_election.cc:304] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.128093 5600 tablet_bootstrap.cc:492] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:14.128335 5990 raft_consensus.cc:2798] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:14.128718 5990 raft_consensus.cc:695] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:14.129423 5990 consensus_queue.cc:237] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.129750 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 152) took 2849 ms. Trace:
W20250110 20:57:14.121266 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 47135.6us
W20250110 20:57:14.138206 5398 leader_election.cc:343] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: cd0a666892d04e97a4b9967e38ab0946
I20250110 20:57:14.138593 5398 leader_election.cc:304] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.139040 5600 tablet_bootstrap.cc:654] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.142315 5979 raft_consensus.cc:2743] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.148960 5110 leader_election.cc:304] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:14.150200 5987 raft_consensus.cc:2798] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:14.150507 5987 raft_consensus.cc:491] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:14.150548 5110 leader_election.cc:304] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:14.150839 5987 raft_consensus.cc:3054] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.150873 6060 consensus_queue.cc:1035] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:14.137665 6048 raft_consensus.cc:2743] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.154153 6066 raft_consensus.cc:2743] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:14.154796 5257 leader_election.cc:343] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.155542 6060 consensus_queue.cc:1035] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:14.161320 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:14.162539 5987 raft_consensus.cc:513] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:14.164363 5257 leader_election.cc:343] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.164718 5257 leader_election.cc:304] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.166281 6060 raft_consensus.cc:2743] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.168001 6055 tablet.cc:1721] T 04fd9785888141abaab655f0ba74efb9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:14.169559 5170 raft_consensus.cc:3054] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:14.171806 5111 leader_election.cc:343] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:14.130219 5595 rpcz_store.cc:276] 0110 20:57:11.280438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.281099 (+ 661us) service_pool.cc:225] Handling call
0110 20:57:13.796223 (+2515124us) ts_tablet_manager.cc:1145] Deleting tablet 491626848ad54be3a18186c65d6dbb60
0110 20:57:13.796262 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.830978 (+ 34716us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.031469 (+200491us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.129699 (+ 98230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18562,"tablet-delete.queue_time_us":2514662}
I20250110 20:57:14.172392 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a09ea54e79485b9892d52d0e04da85" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:14.172852 5987 leader_election.cc:290] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:14.173309 5595 tablet_replica.cc:331] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:14.174113 5595 raft_consensus.cc:2238] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.174574 5595 raft_consensus.cc:2267] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.174425 5110 leader_election.cc:304] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:14.176121 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 491626848ad54be3a18186c65d6dbb60 (table default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]) successfully deleted
I20250110 20:57:14.177333 5990 consensus_queue.cc:1035] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:14.178009 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0b5ba9f38c1493bba01180744f080e1" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:14.178591 5458 raft_consensus.cc:3054] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.142943 5459 raft_consensus.cc:2463] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:14.192545 5255 leader_election.cc:304] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.192296 5592 rpcz_store.cc:276] 0110 20:57:11.967530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.972836 (+ 5306us) service_pool.cc:225] Handling call
0110 20:57:13.696210 (+1723374us) ts_tablet_manager.cc:1145] Deleting tablet 31b65488c884452d876ced1c3d1efb9a
0110 20:57:13.696252 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:13.933472 (+237220us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.018893 (+ 85421us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.069880 (+ 50987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722859}
I20250110 20:57:14.193641 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 31b65488c884452d876ced1c3d1efb9a (table default.Table_d29551acec544d57b14d18fecbf2722e [id=16478fe1cba54c88ba8245c365d5a992]) successfully deleted
I20250110 20:57:14.200011 6060 raft_consensus.cc:2798] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:14.200781 6060 raft_consensus.cc:695] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Becoming Leader. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Running, Role: LEADER
I20250110 20:57:14.201660 6060 consensus_queue.cc:237] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.216645 5592 tablet_replica.cc:331] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:14.218920 5458 raft_consensus.cc:2463] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:14.220963 5600 tablet_bootstrap.cc:492] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:14.221392 5600 ts_tablet_manager.cc:1397] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.002s
I20250110 20:57:14.223495 5600 raft_consensus.cc:357] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.221558 5990 raft_consensus.cc:491] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.224303 5990 raft_consensus.cc:513] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.224575 6059 raft_consensus.cc:2743] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.226080 5990 leader_election.cc:290] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:14.227515 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1ca1670916401ab0a43685df1b6759" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:14.228575 5110 leader_election.cc:304] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:14.229645 6059 raft_consensus.cc:2798] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 20:57:14.229933 5399 leader_election.cc:343] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.230069 6059 raft_consensus.cc:695] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:14.230790 6059 consensus_queue.cc:237] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:14.232004 5398 leader_election.cc:343] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 2c1ca1670916401ab0a43685df1b6759
I20250110 20:57:14.232380 5398 leader_election.cc:304] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.233326 5990 raft_consensus.cc:2743] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.256677 5597 tablet_bootstrap.cc:654] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.257663 5592 raft_consensus.cc:2238] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.266067 5600 raft_consensus.cc:383] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.266341 5600 raft_consensus.cc:738] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.267068 5592 raft_consensus.cc:2267] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.266891 5600 consensus_queue.cc:260] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.270434 5600 ts_tablet_manager.cc:1428] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20250110 20:57:14.239585 6097 tablet.cc:1721] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:14.243892 5595 ts_tablet_manager.cc:1905] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.272042 5600 tablet_bootstrap.cc:492] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:14.277570 5600 tablet_bootstrap.cc:654] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.080487 5594 tablet_bootstrap.cc:654] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.292904 5592 ts_tablet_manager.cc:1905] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.304303 5987 raft_consensus.cc:491] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.304729 5987 raft_consensus.cc:513] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.306492 5987 leader_election.cc:290] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:14.312754 5594 tablet_bootstrap.cc:492] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:14.315197 5773 ts_tablet_manager.cc:1918] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:14.313108 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:14.321625 5594 ts_tablet_manager.cc:1397] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.301s user 0.008s sys 0.001s
I20250110 20:57:14.326896 5773 log.cc:1198] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/19d6eca349514b9fb0fff7cd6230027c
I20250110 20:57:14.342481 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41de354807004e26bfb147f1d420037e" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:14.343205 5319 raft_consensus.cc:2388] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac9003be04f340f3886b5625cc928eb0 in current term 1: Already voted for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in this term.
I20250110 20:57:14.334882 5594 raft_consensus.cc:357] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.346138 5600 tablet_bootstrap.cc:492] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:14.346192 5594 raft_consensus.cc:383] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.346436 5594 raft_consensus.cc:738] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.346493 5600 ts_tablet_manager.cc:1397] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.075s user 0.016s sys 0.009s
I20250110 20:57:14.348950 5600 raft_consensus.cc:357] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.344215 5111 leader_election.cc:304] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:14.353505 5600 raft_consensus.cc:383] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.353673 5773 ts_tablet_manager.cc:1939] T 19d6eca349514b9fb0fff7cd6230027c P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:14.353818 6059 raft_consensus.cc:3054] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.359932 6059 raft_consensus.cc:2743] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.360320 5600 raft_consensus.cc:738] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.360620 5594 consensus_queue.cc:260] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.355546 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 165) took 1537 ms. Trace:
I20250110 20:57:14.361567 5773 rpcz_store.cc:276] 0110 20:57:12.818215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.819499 (+ 1284us) service_pool.cc:225] Handling call
0110 20:57:14.075142 (+1255643us) ts_tablet_manager.cc:1145] Deleting tablet 19d6eca349514b9fb0fff7cd6230027c
0110 20:57:14.075182 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.247202 (+172020us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.314995 (+ 67793us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.355492 (+ 40497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1255095}
I20250110 20:57:14.361752 5600 consensus_queue.cc:260] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.362859 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 164) took 1544 ms. Trace:
I20250110 20:57:14.362823 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 19d6eca349514b9fb0fff7cd6230027c (table default.Table_931826e8e40a4d04a1382a2816608f99 [id=bb54800eb7624f4482ddee01f59494ff]) successfully deleted
I20250110 20:57:14.363137 5773 rpcz_store.cc:276] 0110 20:57:12.817956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.820130 (+ 2174us) service_pool.cc:225] Handling call
0110 20:57:14.362554 (+1542424us) ts_tablet_manager.cc:1145] Deleting tablet a687769f69d844cc8564e61dc324d2b2
0110 20:57:14.362591 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.362811 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541507}
I20250110 20:57:14.363996 5773 tablet_replica.cc:331] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:14.364255 5600 ts_tablet_manager.cc:1428] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.018s user 0.006s sys 0.000s
I20250110 20:57:14.364689 5773 raft_consensus.cc:2238] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
W20250110 20:57:14.365198 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet a687769f69d844cc8564e61dc324d2b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a687769f69d844cc8564e61dc324d2b2 already in progress: creating tablet
I20250110 20:57:14.365468 5773 raft_consensus.cc:2267] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.366398 5600 tablet_bootstrap.cc:492] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:14.368928 5597 tablet_bootstrap.cc:492] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:14.369294 5597 ts_tablet_manager.cc:1397] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.354s user 0.011s sys 0.003s
I20250110 20:57:14.375686 5597 raft_consensus.cc:357] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.376295 5597 raft_consensus.cc:383] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.376533 5597 raft_consensus.cc:738] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.376885 5595 ts_tablet_manager.cc:1918] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:14.380558 5773 ts_tablet_manager.cc:1905] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.380947 5595 log.cc:1198] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/fcc7b1acf4114bf69d801437855760df
I20250110 20:57:14.381079 6097 tablet.cc:1721] T 04fd9785888141abaab655f0ba74efb9 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:14.381671 5595 ts_tablet_manager.cc:1939] T fcc7b1acf4114bf69d801437855760df P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:14.394626 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 149) took 3115 ms. Trace:
I20250110 20:57:14.394994 5595 rpcz_store.cc:276] 0110 20:57:11.279556 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.347162 (+ 67606us) service_pool.cc:225] Handling call
0110 20:57:14.173167 (+2826005us) ts_tablet_manager.cc:1145] Deleting tablet fcc7b1acf4114bf69d801437855760df
0110 20:57:14.173209 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.285095 (+111886us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.376644 (+ 91549us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.394573 (+ 17929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2823314}
I20250110 20:57:14.377053 5597 consensus_queue.cc:260] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.404677 5594 ts_tablet_manager.cc:1428] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.082s user 0.005s sys 0.000s
I20250110 20:57:14.400247 5929 raft_consensus.cc:491] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.406646 5929 raft_consensus.cc:513] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.408383 5929 leader_election.cc:290] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:14.410063 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:14.410246 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet fcc7b1acf4114bf69d801437855760df (table default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]) successfully deleted
I20250110 20:57:14.411358 5595 tablet_replica.cc:331] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
W20250110 20:57:14.411409 5257 leader_election.cc:343] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.412075 5595 raft_consensus.cc:2238] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.412477 5595 raft_consensus.cc:2267] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.380604 5600 tablet_bootstrap.cc:654] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.417358 5773 ts_tablet_manager.cc:1918] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:14.417740 5773 log.cc:1198] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:14.417632 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:14.418817 5255 leader_election.cc:343] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.419178 5255 leader_election.cc:304] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.420009 5929 raft_consensus.cc:2743] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.414011 5595 ts_tablet_manager.cc:1905] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.418370 5773 ts_tablet_manager.cc:1939] T ae5956b1aa17449497f9695a111ebb66 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:14.466737 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 48e53d95ef8645c0bd123138655305ac
I20250110 20:57:14.468696 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e53d95ef8645c0bd123138655305ac with delete_type TABLET_DATA_DELETED (replaced by tablet ba0d83a50ec0479f9673a57370b0e879) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:14.470695 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e53d95ef8645c0bd123138655305ac with delete_type TABLET_DATA_DELETED (replaced by tablet ba0d83a50ec0479f9673a57370b0e879) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:14.471326 4897 catalog_manager.cc:6596] ReplaceTablet: tablet 48e53d95ef8645c0bd123138655305ac deleted and replaced by tablet ba0d83a50ec0479f9673a57370b0e879
I20250110 20:57:14.471688 4897 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36568 (request call id 214) took 1129 ms. Trace:
I20250110 20:57:14.474116 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e53d95ef8645c0bd123138655305ac with delete_type TABLET_DATA_DELETED (replaced by tablet ba0d83a50ec0479f9673a57370b0e879) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:14.478317 5929 raft_consensus.cc:491] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.472010 4897 rpcz_store.cc:276] 0110 20:57:13.342119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:13.668245 (+326126us) service_pool.cc:225] Handling call
0110 20:57:13.672420 (+ 4175us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29dd31 00007f3bfd01fd47 00007f3bfd01fcd2 00007f3bfd01fc01 00007f3bfd01fb7c 00007f3bfd01ee32 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:14.471635 (+799215us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:14.446157 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:14.446491 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:14.446513 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:14.446526 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:14.446790 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:14.446826 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:14.446847 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:14.447129 (+ 282us) tablet.cc:698] Row locks acquired
0110 20:57:14.447143 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:14.447242 (+ 99us) write_op.cc:270] Start()
0110 20:57:14.447317 (+ 75us) write_op.cc:276] Timestamp: P: 1736542634447154 usec, L: 0
0110 20:57:14.447332 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:14.447743 (+ 411us) log.cc:844] Serialized 628 byte log entry
0110 20:57:14.454321 (+ 6578us) op_driver.cc:464] REPLICATION: finished
0110 20:57:14.464616 (+ 10295us) write_op.cc:301] APPLY: starting
0110 20:57:14.464666 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:14.464760 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:14.464859 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:14.465170 (+ 311us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:14.465351 (+ 181us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:14.465394 (+ 43us) write_op.cc:312] APPLY: finished
0110 20:57:14.465579 (+ 185us) log.cc:844] Serialized 29 byte log entry
0110 20:57:14.466133 (+ 554us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.466197 (+ 64us) write_op.cc:454] Released schema lock
0110 20:57:14.466227 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":7484672,"child_traces":[["op",{"apply.queue_time_us":10165,"num_ops":2,"prepare.queue_time_us":867,"prepare.run_cpu_time_us":1803,"prepare.run_wall_time_us":4111,"raft.queue_time_us":115,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":535,"replication_time_us":6919}]]}
I20250110 20:57:14.478736 5929 raft_consensus.cc:513] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.480435 5929 leader_election.cc:290] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:14.482070 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:14.483263 5255 leader_election.cc:343] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.498584 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 163) took 1693 ms. Trace:
I20250110 20:57:14.498909 5773 rpcz_store.cc:276] 0110 20:57:12.817663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.821905 (+ 4242us) service_pool.cc:225] Handling call
0110 20:57:14.363869 (+1541964us) ts_tablet_manager.cc:1145] Deleting tablet ae5956b1aa17449497f9695a111ebb66
0110 20:57:14.363904 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.399198 (+ 35294us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.417143 (+ 17945us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.498523 (+ 81380us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59232,"tablet-delete.queue_time_us":1537443}
I20250110 20:57:14.500046 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 166) took 1524 ms. Trace:
I20250110 20:57:14.500321 5773 rpcz_store.cc:276] 0110 20:57:12.976120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.976755 (+ 635us) service_pool.cc:225] Handling call
0110 20:57:14.499751 (+1522996us) ts_tablet_manager.cc:1145] Deleting tablet f3104928188c489d93bfd499d148fa5d
0110 20:57:14.499789 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.499999 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1501935}
I20250110 20:57:14.501121 5773 tablet_replica.cc:331] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:14.501849 5773 raft_consensus.cc:2238] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.502293 5773 raft_consensus.cc:2267] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.504241 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet ae5956b1aa17449497f9695a111ebb66 (table default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]) successfully deleted
I20250110 20:57:14.505556 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:14.508390 5597 ts_tablet_manager.cc:1428] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.135s user 0.004s sys 0.000s
I20250110 20:57:14.509239 5597 tablet_bootstrap.cc:492] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:14.510807 5773 ts_tablet_manager.cc:1905] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.501353 5595 ts_tablet_manager.cc:1918] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 20:57:14.507261 5257 leader_election.cc:343] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.514065 5595 log.cc:1198] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/5c5b954dbccb45b2a6ab7cc5cceadffb
I20250110 20:57:14.514137 5257 leader_election.cc:304] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.514673 5595 ts_tablet_manager.cc:1939] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:14.515388 5929 raft_consensus.cc:2743] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.516722 5592 ts_tablet_manager.cc:1918] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:14.517074 5592 log.cc:1198] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:14.521677 5594 tablet_bootstrap.cc:492] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:14.527444 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 153) took 2577 ms. Trace:
I20250110 20:57:14.528607 5595 rpcz_store.cc:276] 0110 20:57:11.949668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.950755 (+ 1087us) service_pool.cc:225] Handling call
0110 20:57:14.411079 (+2460324us) ts_tablet_manager.cc:1145] Deleting tablet 5c5b954dbccb45b2a6ab7cc5cceadffb
0110 20:57:14.411256 (+ 177us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.463465 (+ 52209us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.501152 (+ 37687us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.527388 (+ 26236us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2459831}
I20250110 20:57:14.529608 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 5c5b954dbccb45b2a6ab7cc5cceadffb (table default.Table_b89da7c5e86341c39853bbacce851ad4 [id=7833678313fa4c76ad1eb6d19315bb23]) successfully deleted
I20250110 20:57:14.533660 5595 tablet_replica.cc:331] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:14.535411 5595 raft_consensus.cc:2238] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.543001 5595 raft_consensus.cc:2267] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.543498 4899 catalog_manager.cc:5526] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194). New cstate: current_term: 1 leader_uuid: "46005a5920cc4a3697a5c61b1c271384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:14.544870 4899 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:14.545462 5595 ts_tablet_manager.cc:1905] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.545560 4899 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:14.547883 5773 ts_tablet_manager.cc:1918] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:14.550634 5773 log.cc:1198] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/d310cf52aa8c4930a7821117a46dc958
I20250110 20:57:14.551776 4893 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet d1afe41a09954fc8873ea53a330ea03f
I20250110 20:57:14.553537 5773 ts_tablet_manager.cc:1939] T d310cf52aa8c4930a7821117a46dc958 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:14.553884 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet d1afe41a09954fc8873ea53a330ea03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:14.555402 5441 tablet_service.cc:1467] Processing CreateTablet for tablet c72bb739d12244eca51224bc27f0ed1c (DEFAULT_TABLE table=default.Table_054981f06f6f4bd4954b77b6b2ec4436 [id=61d0e26ab8f3495fbdf009038036cbac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.556344 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c72bb739d12244eca51224bc27f0ed1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.558787 5594 tablet_bootstrap.cc:654] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.561060 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdb2ee9419244a9ab939be56b2d7c72 (DEFAULT_TABLE table=default.Table_6fc93557ff4c413284a1e596711e5753 [id=af1b518112b74eb2a7568ddb2e8a1935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.562712 5434 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdb2ee9419244a9ab939be56b2d7c72 (DEFAULT_TABLE table=default.Table_6fc93557ff4c413284a1e596711e5753 [id=af1b518112b74eb2a7568ddb2e8a1935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.563086 5600 tablet_bootstrap.cc:492] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:14.563452 5600 ts_tablet_manager.cc:1397] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.197s user 0.019s sys 0.010s
I20250110 20:57:14.563673 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fdb2ee9419244a9ab939be56b2d7c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.565598 5600 raft_consensus.cc:357] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.566193 5600 raft_consensus.cc:383] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.566438 5600 raft_consensus.cc:738] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.567303 5292 tablet_service.cc:1467] Processing CreateTablet for tablet c72bb739d12244eca51224bc27f0ed1c (DEFAULT_TABLE table=default.Table_054981f06f6f4bd4954b77b6b2ec4436 [id=61d0e26ab8f3495fbdf009038036cbac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.567214 5600 consensus_queue.cc:260] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.567966 5597 tablet_bootstrap.cc:654] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.571348 5600 ts_tablet_manager.cc:1428] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20250110 20:57:14.574198 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet d1afe41a09954fc8873ea53a330ea03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:14.575773 5594 tablet_bootstrap.cc:492] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:14.576130 5594 ts_tablet_manager.cc:1397] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.006s
I20250110 20:57:14.576493 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 3fdb2ee9419244a9ab939be56b2d7c72 (DEFAULT_TABLE table=default.Table_6fc93557ff4c413284a1e596711e5753 [id=af1b518112b74eb2a7568ddb2e8a1935]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.577478 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fdb2ee9419244a9ab939be56b2d7c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.578878 5594 raft_consensus.cc:357] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.579507 5594 raft_consensus.cc:383] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.579728 5594 raft_consensus.cc:738] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.580386 5594 consensus_queue.cc:260] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.582194 5146 tablet_service.cc:1467] Processing CreateTablet for tablet c72bb739d12244eca51224bc27f0ed1c (DEFAULT_TABLE table=default.Table_054981f06f6f4bd4954b77b6b2ec4436 [id=61d0e26ab8f3495fbdf009038036cbac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.583149 5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c72bb739d12244eca51224bc27f0ed1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.586820 5594 ts_tablet_manager.cc:1428] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.010s user 0.004s sys 0.002s
I20250110 20:57:14.588461 5594 tablet_bootstrap.cc:492] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:14.590691 5597 tablet_bootstrap.cc:492] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:14.591073 5597 ts_tablet_manager.cc:1397] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.082s user 0.008s sys 0.001s
I20250110 20:57:14.593086 5597 raft_consensus.cc:357] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.593739 5597 raft_consensus.cc:383] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.593993 5597 raft_consensus.cc:738] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.593931 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]) successfully deleted
I20250110 20:57:14.594795 5594 tablet_bootstrap.cc:654] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:14.594794 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 48e53d95ef8645c0bd123138655305ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e53d95ef8645c0bd123138655305ac already in progress: creating tablet
I20250110 20:57:14.594640 5597 consensus_queue.cc:260] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.596952 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet d1afe41a09954fc8873ea53a330ea03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:14.597637 4893 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 215) took 1019 ms. Trace:
I20250110 20:57:14.605207 5594 tablet_bootstrap.cc:492] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:14.605599 5594 ts_tablet_manager.cc:1397] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20250110 20:57:14.607733 5594 raft_consensus.cc:357] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.608345 5594 raft_consensus.cc:383] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.608578 5594 raft_consensus.cc:738] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.610316 5597 ts_tablet_manager.cc:1428] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20250110 20:57:14.612887 5594 consensus_queue.cc:260] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.615935 5594 ts_tablet_manager.cc:1428] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250110 20:57:14.616683 5594 tablet_bootstrap.cc:492] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:14.622485 5594 tablet_bootstrap.cc:654] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.622627 5597 tablet_bootstrap.cc:492] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:14.622964 5595 ts_tablet_manager.cc:1918] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:14.623296 5595 log.cc:1198] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/c951d5ca47e8409db646b43bb04a7034
I20250110 20:57:14.623831 5595 ts_tablet_manager.cc:1939] T c951d5ca47e8409db646b43bb04a7034 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:14.638586 5594 tablet_bootstrap.cc:492] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:14.638962 5594 ts_tablet_manager.cc:1397] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.005s
I20250110 20:57:14.640954 4892 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 04fd9785888141abaab655f0ba74efb9 (table default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d]). Expected version 1 got 0
I20250110 20:57:14.640985 5594 raft_consensus.cc:357] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.641611 5594 raft_consensus.cc:383] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.642004 5594 raft_consensus.cc:738] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.642504 5597 tablet_bootstrap.cc:654] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.642552 5594 consensus_queue.cc:260] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.644518 5594 ts_tablet_manager.cc:1428] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20250110 20:57:14.645303 5594 tablet_bootstrap.cc:492] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:14.650383 5594 tablet_bootstrap.cc:654] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.668367 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet fde470279464496aa7efdee3800754f3 (table default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]). Expected version 1 got 0
I20250110 20:57:14.682020 5773 tablet_replica.cc:331] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:14.683171 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fdb2ee9419244a9ab939be56b2d7c72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.689885 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c72bb739d12244eca51224bc27f0ed1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.690191 5600 tablet_bootstrap.cc:492] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:14.690893 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 154) took 2741 ms. Trace:
I20250110 20:57:14.691211 5595 rpcz_store.cc:276] 0110 20:57:11.949969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.951327 (+ 1358us) service_pool.cc:225] Handling call
0110 20:57:14.533525 (+2582198us) ts_tablet_manager.cc:1145] Deleting tablet c951d5ca47e8409db646b43bb04a7034
0110 20:57:14.533563 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.588827 (+ 55264us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.622770 (+ 33943us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.690837 (+ 68067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2581709}
I20250110 20:57:14.692160 5595 tablet_replica.cc:331] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:14.692802 5595 raft_consensus.cc:2238] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.693230 5595 raft_consensus.cc:2267] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.695953 5600 tablet_bootstrap.cc:654] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.699657 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:14.700218 5592 ts_tablet_manager.cc:1939] T ae5956b1aa17449497f9695a111ebb66 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:14.706182 5595 ts_tablet_manager.cc:1905] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.718086 5773 raft_consensus.cc:2238] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.718572 5773 raft_consensus.cc:2267] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.697642 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet c951d5ca47e8409db646b43bb04a7034 (table default.Table_2a36a6d3c1c843eba552b6fd97984099 [id=723ca40b46694de9802850cb89dca66f]) successfully deleted
I20250110 20:57:14.597994 4893 rpcz_store.cc:276] 0110 20:57:13.578668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:13.790854 (+212186us) service_pool.cc:225] Handling call
0110 20:57:13.792063 (+ 1209us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:13.792593 (+ 530us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:13.792701 (+ 108us) catalog_manager.cc:2762] Locking tablets
0110 20:57:14.523674 (+730973us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:14.540224 (+ 16550us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:14.551606 (+ 11382us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:14.551705 (+ 99us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:14.597571 (+ 45866us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:14.525872 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:14.526218 (+ 346us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:14.526240 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:14.526252 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:14.526551 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:14.526587 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:14.526608 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:14.526839 (+ 231us) tablet.cc:698] Row locks acquired
0110 20:57:14.526852 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:14.526953 (+ 101us) write_op.cc:270] Start()
0110 20:57:14.527036 (+ 83us) write_op.cc:276] Timestamp: P: 1736542634526855 usec, L: 0
0110 20:57:14.527052 (+ 16us) op_driver.cc:348] REPLICATION: starting
0110 20:57:14.527547 (+ 495us) log.cc:844] Serialized 731 byte log entry
0110 20:57:14.536041 (+ 8494us) op_driver.cc:464] REPLICATION: finished
0110 20:57:14.538630 (+ 2589us) write_op.cc:301] APPLY: starting
0110 20:57:14.538684 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:14.538778 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:14.538861 (+ 83us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:14.539092 (+ 231us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:14.539216 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:14.539257 (+ 41us) write_op.cc:312] APPLY: finished
0110 20:57:14.539415 (+ 158us) log.cc:844] Serialized 29 byte log entry
0110 20:57:14.539951 (+ 536us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.540016 (+ 65us) write_op.cc:454] Released schema lock
0110 20:57:14.540057 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:14.540478 (+ 421us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.540502 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":2455,"apply.run_cpu_time_us":1210,"apply.run_wall_time_us":1758,"num_ops":2,"prepare.queue_time_us":247,"prepare.run_cpu_time_us":1901,"prepare.run_wall_time_us":4431,"raft.queue_time_us":72,"raft.run_cpu_time_us":613,"raft.run_wall_time_us":612,"replication_time_us":8894}]]}
I20250110 20:57:14.727633 5145 tablet_service.cc:1467] Processing CreateTablet for tablet f482cd146ca34f9e95b2361e9c0a985a (DEFAULT_TABLE table=default.Table_ec0889bc4a894c00bcc098b12fa053fd [id=4449f4635e6d44b6ba07a80fe9a4fa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.728720 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f482cd146ca34f9e95b2361e9c0a985a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.740600 5302 tablet_service.cc:1467] Processing CreateTablet for tablet f482cd146ca34f9e95b2361e9c0a985a (DEFAULT_TABLE table=default.Table_ec0889bc4a894c00bcc098b12fa053fd [id=4449f4635e6d44b6ba07a80fe9a4fa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.740710 5443 tablet_service.cc:1467] Processing CreateTablet for tablet f482cd146ca34f9e95b2361e9c0a985a (DEFAULT_TABLE table=default.Table_ec0889bc4a894c00bcc098b12fa053fd [id=4449f4635e6d44b6ba07a80fe9a4fa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.741587 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f482cd146ca34f9e95b2361e9c0a985a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.741706 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f482cd146ca34f9e95b2361e9c0a985a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.746017 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]). Expected version 1 got 0
I20250110 20:57:14.746927 4900 catalog_manager.cc:5526] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:14.748095 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:14.753933 5595 ts_tablet_manager.cc:1918] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:14.754302 5595 log.cc:1198] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:14.754946 5595 ts_tablet_manager.cc:1939] T ae5956b1aa17449497f9695a111ebb66 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:14.757018 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 164) took 2019 ms. Trace:
I20250110 20:57:14.763154 5773 ts_tablet_manager.cc:1905] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.772001 5979 raft_consensus.cc:491] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.772445 5979 raft_consensus.cc:513] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.774276 5979 leader_election.cc:290] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:14.778501 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:14.781791 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 356a19079fb94aa19e2b4c85ce616f66 (DEFAULT_TABLE table=default.Table_8d1a07d9e4cf46ec81956254772651a4 [id=83af0c08577b43aeacf6287ace26bed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.782173 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 356a19079fb94aa19e2b4c85ce616f66 (DEFAULT_TABLE table=default.Table_8d1a07d9e4cf46ec81956254772651a4 [id=83af0c08577b43aeacf6287ace26bed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.782774 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a19079fb94aa19e2b4c85ce616f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.783138 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a19079fb94aa19e2b4c85ce616f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.784794 5155 tablet_service.cc:1467] Processing CreateTablet for tablet a68d4b44975b436fb1b4a9ee6ff290d2 (DEFAULT_TABLE table=default.Table_156534e2fa0e4fa7bd3eb498d05a4626 [id=9a3db504db8449efb89996c55376ab19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.785779 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68d4b44975b436fb1b4a9ee6ff290d2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:14.792390 5399 leader_election.cc:343] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.757359 5592 rpcz_store.cc:276] 0110 20:57:12.737213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.737348 (+ 135us) service_pool.cc:225] Handling call
0110 20:57:14.216494 (+1479146us) ts_tablet_manager.cc:1145] Deleting tablet ae5956b1aa17449497f9695a111ebb66
0110 20:57:14.216543 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.347807 (+131264us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.516525 (+168718us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.756954 (+240429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1478641}
I20250110 20:57:14.805313 5597 tablet_bootstrap.cc:492] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:14.805536 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet ae5956b1aa17449497f9695a111ebb66 (table default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]) successfully deleted
I20250110 20:57:14.805696 5597 ts_tablet_manager.cc:1397] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.195s user 0.018s sys 0.026s
I20250110 20:57:14.806540 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 165) took 2053 ms. Trace:
I20250110 20:57:14.806876 5592 rpcz_store.cc:276] 0110 20:57:12.752871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.752998 (+ 127us) service_pool.cc:225] Handling call
0110 20:57:14.806233 (+2053235us) ts_tablet_manager.cc:1145] Deleting tablet a687769f69d844cc8564e61dc324d2b2
0110 20:57:14.806274 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.806491 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2052239}
I20250110 20:57:14.807605 5597 raft_consensus.cc:357] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:14.807989 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet a687769f69d844cc8564e61dc324d2b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a687769f69d844cc8564e61dc324d2b2 already in progress: creating tablet
I20250110 20:57:14.808203 5597 raft_consensus.cc:383] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.808413 5597 raft_consensus.cc:738] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.808866 5592 tablet_replica.cc:331] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:14.809613 5592 raft_consensus.cc:2238] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.816941 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 356a19079fb94aa19e2b4c85ce616f66 (DEFAULT_TABLE table=default.Table_8d1a07d9e4cf46ec81956254772651a4 [id=83af0c08577b43aeacf6287ace26bed2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.829674 5597 consensus_queue.cc:260] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.831037 6059 raft_consensus.cc:491] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.831423 6059 raft_consensus.cc:513] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.832376 6047 raft_consensus.cc:491] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.832791 6047 raft_consensus.cc:513] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.748915 4900 catalog_manager.cc:5526] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:14.833761 5929 raft_consensus.cc:491] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.834290 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:14.834149 5929 raft_consensus.cc:513] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.834898 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
I20250110 20:57:14.835539 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:14.835862 5929 leader_election.cc:290] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:14.836140 5318 raft_consensus.cc:2463] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:14.827919 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 356a19079fb94aa19e2b4c85ce616f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.838655 6047 leader_election.cc:290] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:14.840916 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:14.841266 6059 leader_election.cc:290] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:14.842101 5257 leader_election.cc:343] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.843545 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e53d95ef8645c0bd123138655305ac" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:14.844534 5257 leader_election.cc:343] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:14.846896 5111 leader_election.cc:343] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: c951d5ca47e8409db646b43bb04a7034
I20250110 20:57:14.849710 5301 tablet_service.cc:1467] Processing CreateTablet for tablet a68d4b44975b436fb1b4a9ee6ff290d2 (DEFAULT_TABLE table=default.Table_156534e2fa0e4fa7bd3eb498d05a4626 [id=9a3db504db8449efb89996c55376ab19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.850752 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68d4b44975b436fb1b4a9ee6ff290d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.837145 5398 leader_election.cc:304] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.842983 5979 raft_consensus.cc:491] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:14.861857 5979 raft_consensus.cc:513] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.863772 5979 leader_election.cc:290] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:14.864745 5979 raft_consensus.cc:2798] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:14.865073 5979 raft_consensus.cc:491] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:14.865381 5979 raft_consensus.cc:3054] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.869031 5592 raft_consensus.cc:2267] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:14.851722 5441 tablet_service.cc:1467] Processing CreateTablet for tablet a68d4b44975b436fb1b4a9ee6ff290d2 (DEFAULT_TABLE table=default.Table_156534e2fa0e4fa7bd3eb498d05a4626 [id=9a3db504db8449efb89996c55376ab19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.870100 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 161) took 2116 ms. Trace:
I20250110 20:57:14.870354 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a68d4b44975b436fb1b4a9ee6ff290d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.870718 5592 ts_tablet_manager.cc:1905] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.870460 5595 rpcz_store.cc:276] 0110 20:57:12.753626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.753739 (+ 113us) service_pool.cc:225] Handling call
0110 20:57:14.692033 (+1938294us) ts_tablet_manager.cc:1145] Deleting tablet ae5956b1aa17449497f9695a111ebb66
0110 20:57:14.692068 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.721454 (+ 29386us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.753721 (+ 32267us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.870036 (+116315us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62518,"tablet-delete.queue_time_us":1937757}
I20250110 20:57:14.871706 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet ae5956b1aa17449497f9695a111ebb66 (table default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]) successfully deleted
I20250110 20:57:14.873697 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 162) took 2119 ms. Trace:
I20250110 20:57:14.874024 5595 rpcz_store.cc:276] 0110 20:57:12.753955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.754361 (+ 406us) service_pool.cc:225] Handling call
0110 20:57:14.872914 (+2118553us) ts_tablet_manager.cc:1145] Deleting tablet a687769f69d844cc8564e61dc324d2b2
0110 20:57:14.872953 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.873167 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2118079}
I20250110 20:57:14.874295 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6914e258314423c859eb31e85853274" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:14.877827 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet a687769f69d844cc8564e61dc324d2b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a687769f69d844cc8564e61dc324d2b2 already in progress: creating tablet
I20250110 20:57:14.879628 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 76bedccd828042948a702fe840788765 (DEFAULT_TABLE table=default.Table_31bd404aec6f4df489bae22a80054bfc [id=bef02f1acb0e4268882c92bc6c3618b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.881214 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bedccd828042948a702fe840788765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.884840 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:14.885421 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 76bedccd828042948a702fe840788765 (DEFAULT_TABLE table=default.Table_31bd404aec6f4df489bae22a80054bfc [id=bef02f1acb0e4268882c92bc6c3618b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.886379 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bedccd828042948a702fe840788765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.887945 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 76bedccd828042948a702fe840788765 (DEFAULT_TABLE table=default.Table_31bd404aec6f4df489bae22a80054bfc [id=bef02f1acb0e4268882c92bc6c3618b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.888664 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e53d95ef8645c0bd123138655305ac" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:14.888973 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bedccd828042948a702fe840788765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.889997 5595 tablet_replica.cc:331] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:14.890769 5595 raft_consensus.cc:2238] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:14.891242 5595 raft_consensus.cc:2267] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:14.891988 5111 consensus_peers.cc:487] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:14.892827 5255 leader_election.cc:343] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.893276 5255 leader_election.cc:304] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.894116 6130 raft_consensus.cc:2743] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:14.894977 5255 leader_election.cc:343] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.895397 5255 leader_election.cc:304] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.895996 6130 raft_consensus.cc:2743] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:14.897102 5399 leader_election.cc:343] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.897763 5600 tablet_bootstrap.cc:492] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:14.898133 5600 ts_tablet_manager.cc:1397] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.208s user 0.019s sys 0.017s
I20250110 20:57:14.900205 5600 raft_consensus.cc:357] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.900799 5600 raft_consensus.cc:383] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.901031 5600 raft_consensus.cc:738] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.904382 5597 ts_tablet_manager.cc:1428] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.098s user 0.004s sys 0.001s
I20250110 20:57:14.892846 5595 ts_tablet_manager.cc:1905] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:14.904232 5979 raft_consensus.cc:513] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.908494 5597 tablet_bootstrap.cc:492] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:14.910075 5979 leader_election.cc:290] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:14.911305 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:14.912569 5399 leader_election.cc:343] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:14.932724 5146 tablet_service.cc:1467] Processing CreateTablet for tablet 95b52e115a8e4bae95da0ef9d5afff8c (DEFAULT_TABLE table=default.Table_2a36a6d3c1c843eba552b6fd97984099 [id=723ca40b46694de9802850cb89dca66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:14.932603 5600 consensus_queue.cc:260] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:14.937000 5600 ts_tablet_manager.cc:1428] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20250110 20:57:14.937865 5600 tablet_bootstrap.cc:492] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:14.952764 5398 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 115687us
I20250110 20:57:14.957790 5594 tablet_bootstrap.cc:492] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:14.958194 5594 ts_tablet_manager.cc:1397] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.313s user 0.029s sys 0.042s
I20250110 20:57:14.958184 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:14.958725 5323 raft_consensus.cc:3054] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:14.958772 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:14.960376 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:14.961052 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:14.961822 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]). Expected version 1 got 0
I20250110 20:57:14.962488 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
I20250110 20:57:14.965569 5594 raft_consensus.cc:357] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.966168 5594 raft_consensus.cc:383] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:14.966411 5594 raft_consensus.cc:738] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:14.966975 5594 consensus_queue.cc:260] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:14.968683 5323 raft_consensus.cc:2463] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:14.968972 5594 ts_tablet_manager.cc:1428] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
W20250110 20:57:14.969949 5398 leader_election.cc:343] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: c6914e258314423c859eb31e85853274
I20250110 20:57:14.970400 5398 leader_election.cc:304] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.972178 5398 leader_election.cc:304] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:14.975865 5595 ts_tablet_manager.cc:1918] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:14.976218 5595 log.cc:1198] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/19d6eca349514b9fb0fff7cd6230027c
I20250110 20:57:14.976888 5595 ts_tablet_manager.cc:1939] T 19d6eca349514b9fb0fff7cd6230027c P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:14.983625 6137 raft_consensus.cc:2743] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:14.986240 6137 raft_consensus.cc:2798] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:14.987171 5600 tablet_bootstrap.cc:654] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:14.987661 6137 raft_consensus.cc:695] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:14.991427 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 163) took 2213 ms. Trace:
I20250110 20:57:15.010684 5594 tablet_bootstrap.cc:492] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:15.014189 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 9633eb4580c04f11a48ff5055e257c50 (DEFAULT_TABLE table=default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.015058 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9633eb4580c04f11a48ff5055e257c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.011334 5434 tablet_service.cc:1467] Processing CreateTablet for tablet 95b52e115a8e4bae95da0ef9d5afff8c (DEFAULT_TABLE table=default.Table_2a36a6d3c1c843eba552b6fd97984099 [id=723ca40b46694de9802850cb89dca66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.012501 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 95b52e115a8e4bae95da0ef9d5afff8c (DEFAULT_TABLE table=default.Table_2a36a6d3c1c843eba552b6fd97984099 [id=723ca40b46694de9802850cb89dca66f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.018180 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b52e115a8e4bae95da0ef9d5afff8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.018496 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b52e115a8e4bae95da0ef9d5afff8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.014655 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 9633eb4580c04f11a48ff5055e257c50 (DEFAULT_TABLE table=default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.021950 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9633eb4580c04f11a48ff5055e257c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.024102 5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b52e115a8e4bae95da0ef9d5afff8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.024084 5299 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1330e66fc14b4da9e4a4669943360c (DEFAULT_TABLE table=default.Table_03ca54bede3b459497f750a83fe9e117 [id=b78ad0741c774ec487fd064cbaaf9406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.024969 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1330e66fc14b4da9e4a4669943360c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.026440 4893 catalog_manager.cc:5526] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194). New cstate: current_term: 1 leader_uuid: "46005a5920cc4a3697a5c61b1c271384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:15.028064 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet fde470279464496aa7efdee3800754f3 (table default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]). Expected version 1 got 0
I20250110 20:57:15.028440 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1330e66fc14b4da9e4a4669943360c (DEFAULT_TABLE table=default.Table_03ca54bede3b459497f750a83fe9e117 [id=b78ad0741c774ec487fd064cbaaf9406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.028738 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:15.029197 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:15.029374 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1330e66fc14b4da9e4a4669943360c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.029824 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:15.030416 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:15.030987 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:15.032090 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]). Expected version 1 got 0
I20250110 20:57:15.032855 4891 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet e27e43461251413b97721323cd935374
I20250110 20:57:15.034142 4891 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 206) took 2079 ms. Trace:
I20250110 20:57:15.034422 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e43461251413b97721323cd935374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.034431 4891 rpcz_store.cc:276] 0110 20:57:12.955073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.955195 (+ 122us) service_pool.cc:225] Handling call
0110 20:57:13.049060 (+ 93865us) spinlock_profiling.cc:243] Waited 91.2 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b616 00007f3bfd003b44 00007f3bfd003ad2 00007f3bfd003a01 00007f3bfd00397c 00007f3bfd002c32 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:13.050077 (+ 1017us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:13.051074 (+ 997us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:13.051204 (+ 130us) catalog_manager.cc:2762] Locking tablets
0110 20:57:13.080486 (+ 29282us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:13.534158 (+453672us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:13.534371 (+ 213us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:15.032771 (+1498400us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:15.034091 (+ 1320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:13.494913 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:13.495258 (+ 345us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:13.495281 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:13.495294 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:13.495575 (+ 281us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:13.495614 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:13.495637 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:13.495887 (+ 250us) tablet.cc:698] Row locks acquired
0110 20:57:13.495901 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:13.495996 (+ 95us) write_op.cc:270] Start()
0110 20:57:13.496075 (+ 79us) write_op.cc:276] Timestamp: P: 1736542633495901 usec, L: 0
0110 20:57:13.496091 (+ 16us) op_driver.cc:348] REPLICATION: starting
0110 20:57:13.496554 (+ 463us) log.cc:844] Serialized 796 byte log entry
0110 20:57:13.532022 (+ 35468us) op_driver.cc:464] REPLICATION: finished
0110 20:57:13.532338 (+ 316us) write_op.cc:301] APPLY: starting
0110 20:57:13.532386 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:13.532486 (+ 100us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:13.532599 (+ 113us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:13.532817 (+ 218us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:13.532942 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:13.532982 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:13.533140 (+ 158us) log.cc:844] Serialized 29 byte log entry
0110 20:57:13.533276 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:13.533340 (+ 64us) write_op.cc:454] Released schema lock
0110 20:57:13.533368 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:13.549380 (+ 16012us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:13.559363 (+ 9983us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":209765760,"child_traces":[["op",{"apply.queue_time_us":196,"apply.run_cpu_time_us":1211,"apply.run_wall_time_us":1211,"num_ops":2,"prepare.queue_time_us":399094,"prepare.run_cpu_time_us":1981,"prepare.run_wall_time_us":54451,"raft.queue_time_us":377,"raft.run_cpu_time_us":704,"raft.run_wall_time_us":816,"replication_time_us":35848}]]}
I20250110 20:57:15.038599 5979 consensus_queue.cc:1035] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:15.039299 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e43461251413b97721323cd935374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.039247 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_634a503e9332481eb24fe3aa1a7c37a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:15.040634 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_634a503e9332481eb24fe3aa1a7c37a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:15.052892 4886 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 1ac07ac6fc9f477ba7f9accc5533be39 (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 1ac07ac6fc9f477ba7f9accc5533be39 consensus state has no leader
I20250110 20:57:14.996876 5773 ts_tablet_manager.cc:1918] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:15.015708 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 9633eb4580c04f11a48ff5055e257c50 (DEFAULT_TABLE table=default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.017927 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 5c1330e66fc14b4da9e4a4669943360c (DEFAULT_TABLE table=default.Table_03ca54bede3b459497f750a83fe9e117 [id=b78ad0741c774ec487fd064cbaaf9406]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.060075 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c1330e66fc14b4da9e4a4669943360c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.066445 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9633eb4580c04f11a48ff5055e257c50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:14.977795 5597 tablet_bootstrap.cc:654] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.078895 5600 tablet_bootstrap.cc:492] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:15.079304 5600 ts_tablet_manager.cc:1397] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.142s user 0.010s sys 0.000s
I20250110 20:57:15.078429 5773 log.cc:1198] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/d1afe41a09954fc8873ea53a330ea03f
I20250110 20:57:15.083783 5595 rpcz_store.cc:276] 0110 20:57:12.777914 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.778039 (+ 125us) service_pool.cc:225] Handling call
0110 20:57:14.889867 (+2111828us) ts_tablet_manager.cc:1145] Deleting tablet 19d6eca349514b9fb0fff7cd6230027c
0110 20:57:14.889905 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.935257 (+ 45352us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.975659 (+ 40402us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.991371 (+ 15712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111325}
I20250110 20:57:15.045148 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e43461251413b97721323cd935374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.085628 5773 ts_tablet_manager.cc:1939] T d1afe41a09954fc8873ea53a330ea03f P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:15.086545 4891 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36568 (request call id 219) took 1312 ms. Trace:
I20250110 20:57:15.090337 4897 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_b5ff248226c7469494af3086db1957fe [id=b4a25185aa624d20a27bb31f0038436d] alter complete (version 1)
I20250110 20:57:15.090895 6129 raft_consensus.cc:491] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.091336 6129 raft_consensus.cc:513] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:15.094642 5399 leader_election.cc:343] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: d1afe41a09954fc8873ea53a330ea03f
I20250110 20:57:15.105711 5979 consensus_queue.cc:1035] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:15.012032 6137 consensus_queue.cc:237] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.097751 5600 raft_consensus.cc:357] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.125974 5600 raft_consensus.cc:383] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.126307 5600 raft_consensus.cc:738] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.117827 5299 tablet_service.cc:1514] Processing DeleteTablet for tablet 491626848ad54be3a18186c65d6dbb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.127005 5600 consensus_queue.cc:260] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.127581 5148 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d634add82994f74960cd2b8373a4dc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 33db8a2fdd9c4fde985345604fd04215) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.129351 5600 ts_tablet_manager.cc:1428] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20250110 20:57:15.130153 5600 tablet_bootstrap.cc:492] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:15.133452 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 961fef96e9124aafb1c51e287f603b64 with delete_type TABLET_DATA_DELETED (replaced by tablet f2cbb46ea11c4e7d8bc5993ccdc52247) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.133978 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.134830 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 164) took 2214 ms. Trace:
I20250110 20:57:15.135105 5595 rpcz_store.cc:276] 0110 20:57:12.920636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.920766 (+ 130us) service_pool.cc:225] Handling call
0110 20:57:15.134526 (+2213760us) ts_tablet_manager.cc:1145] Deleting tablet f3104928188c489d93bfd499d148fa5d
0110 20:57:15.134565 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:15.134781 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2213239}
I20250110 20:57:15.136082 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 477b09061a4a4f278720168872a83258 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac07ac6fc9f477ba7f9accc5533be39) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.141167 6130 raft_consensus.cc:491] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.141585 6130 raft_consensus.cc:513] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.143394 6130 leader_election.cc:290] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:15.146271 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet d310cf52aa8c4930a7821117a46dc958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4aaa42bc5ab042e79baee1ea838851eb) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.147595 5600 tablet_bootstrap.cc:654] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.148207 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet d9776df98cb54be38a521852634490e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.149097 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6eca349514b9fb0fff7cd6230027c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.149964 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.150771 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b65488c884452d876ced1c3d1efb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.125562 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.151573 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5956b1aa17449497f9695a111ebb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28befa048244309a06f69b8a3b4b8f) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.152405 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet f1fafed70adb496d97acfd5fea356292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.149964 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5956b1aa17449497f9695a111ebb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28befa048244309a06f69b8a3b4b8f) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:14.996217 5592 ts_tablet_manager.cc:1918] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:15.153343 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5956b1aa17449497f9695a111ebb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28befa048244309a06f69b8a3b4b8f) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.153618 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 637a39e4264b40dd9480dc13e37359d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.158345 5595 tablet_replica.cc:331] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:15.160681 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.169489 5595 raft_consensus.cc:2238] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:15.173023 5595 raft_consensus.cc:2267] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:15.177356 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 2967c06715e241dcbbe784eed2373704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.177618 5156 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 with delete_type TABLET_DATA_DELETED (replaced by tablet 371fb7408f0149c8b00659e09ebaf48a) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.177820 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet da24904dece04a6194790054f6d7153a with delete_type TABLET_DATA_DELETED (replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.177639 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 with delete_type TABLET_DATA_DELETED (replaced by tablet 371fb7408f0149c8b00659e09ebaf48a) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.178004 5597 tablet_bootstrap.cc:492] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:15.179040 5597 ts_tablet_manager.cc:1397] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.271s user 0.009s sys 0.000s
I20250110 20:57:15.186477 5595 ts_tablet_manager.cc:1905] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.181221 5597 raft_consensus.cc:357] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.193614 5597 raft_consensus.cc:383] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.193879 5597 raft_consensus.cc:738] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.096068 4894 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_838d31fde06b486f86d7870e9f580912 [id=95036a214c1740d4b8391e06f7110688] alter complete (version 1)
I20250110 20:57:15.196883 5597 consensus_queue.cc:260] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:15.098768 4896 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 56) took 2838 ms (client timeout 1000 ms). Trace:
W20250110 20:57:15.197583 4896 rpcz_store.cc:269] 0110 20:57:12.260402 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.260550 (+ 148us) service_pool.cc:225] Handling call
0110 20:57:12.540304 (+279754us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:15.098697 (+2558393us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":635750144}
W20250110 20:57:15.199077 5111 consensus_peers.cc:487] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0b5ba9f38c1493bba01180744f080e1. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:15.199126 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f70bf3182e2a4bcbafbc98e2977b4f2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:15.201172 5597 ts_tablet_manager.cc:1428] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
W20250110 20:57:15.202050 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f70bf3182e2a4bcbafbc98e2977b4f2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:15.204928 4894 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 48) took 3208 ms (client timeout 1000 ms). Trace:
W20250110 20:57:15.205273 4894 rpcz_store.cc:269] 0110 20:57:11.996733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.996856 (+ 123us) service_pool.cc:225] Handling call
0110 20:57:15.204867 (+3208011us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:13.149917 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:13.150266 (+ 349us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:13.150288 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:13.150301 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:13.152335 (+ 2034us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:13.152374 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:13.152397 (+ 23us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:13.152907 (+ 510us) tablet.cc:698] Row locks acquired
0110 20:57:13.152920 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:13.153024 (+ 104us) write_op.cc:270] Start()
0110 20:57:13.153101 (+ 77us) write_op.cc:276] Timestamp: P: 1736542633152974 usec, L: 0
0110 20:57:13.153115 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:13.153582 (+ 467us) log.cc:844] Serialized 356 byte log entry
0110 20:57:13.365692 (+212110us) op_driver.cc:464] REPLICATION: finished
0110 20:57:13.367444 (+ 1752us) write_op.cc:301] APPLY: starting
0110 20:57:13.367502 (+ 58us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:13.367593 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:13.367684 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:13.367853 (+ 169us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:13.367989 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:13.368030 (+ 41us) write_op.cc:312] APPLY: finished
0110 20:57:13.368202 (+ 172us) log.cc:844] Serialized 23 byte log entry
0110 20:57:13.368341 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:13.368400 (+ 59us) write_op.cc:454] Released schema lock
0110 20:57:13.368429 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:13.368646 (+ 217us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:13.368668 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":1607,"apply.run_cpu_time_us":1171,"apply.run_wall_time_us":1172,"num_ops":1,"prepare.queue_time_us":361259,"prepare.run_cpu_time_us":1745,"prepare.run_wall_time_us":4740,"replication_time_us":212473}]]}
I20250110 20:57:15.208925 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_3c32fcd3cede42049db2626e1952ecb4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:15.210453 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c32fcd3cede42049db2626e1952ecb4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:15.134068 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 19d6eca349514b9fb0fff7cd6230027c (table default.Table_931826e8e40a4d04a1382a2816608f99 [id=bb54800eb7624f4482ddee01f59494ff]) successfully deleted
W20250110 20:57:15.210960 4886 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 76885.9us
I20250110 20:57:15.125910 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.170588 5435 tablet_service.cc:1514] Processing DeleteTablet for tablet da24904dece04a6194790054f6d7153a with delete_type TABLET_DATA_DELETED (replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.171121 5433 tablet_service.cc:1514] Processing DeleteTablet for tablet 2967c06715e241dcbbe784eed2373704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.190941 5594 tablet_bootstrap.cc:700] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent opening tablet: real 0.174s user 0.004s sys 0.000s
W20250110 20:57:15.222689 5398 consensus_peers.cc:487] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:15.171316 5432 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.225715 5436 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:15.155059 4892 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 39) took 2028 ms (client timeout 1000 ms). Trace:
W20250110 20:57:15.226809 4892 rpcz_store.cc:269] 0110 20:57:13.126717 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:13.372343 (+245626us) service_pool.cc:225] Handling call
0110 20:57:13.387339 (+ 14996us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:15.154999 (+1767660us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11574,"reported_tablets":14,"spinlock_wait_cycles":28676480}
I20250110 20:57:15.228600 4892 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_0c447ed02fd84f509e75155308d4a9c0" } new_table_name: "default.Table_1709c1058a134005a1475580de97779b" modify_external_catalogs: true
I20250110 20:57:15.230103 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1afe41a09954fc8873ea53a330ea03f" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:15.230664 5323 raft_consensus.cc:2463] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
W20250110 20:57:15.161648 4895 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 38) took 3323 ms (client timeout 1000 ms). Trace:
W20250110 20:57:15.231858 4895 rpcz_store.cc:269] 0110 20:57:11.838287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:11.838376 (+ 89us) service_pool.cc:225] Handling call
0110 20:57:11.898985 (+ 60609us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:15.161587 (+3262602us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7984,"reported_tablets":14,"spinlock_wait_cycles":123272064}
I20250110 20:57:15.232839 5398 leader_election.cc:304] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:15.233021 6059 consensus_queue.cc:1035] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:15.233625 6143 raft_consensus.cc:2798] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:15.233911 6143 raft_consensus.cc:491] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:15.233927 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_6fc93557ff4c413284a1e596711e5753" } new_table_name: "default.Table_a23e7391e6744a67a193c12dc0ca20c5" modify_external_catalogs: true
I20250110 20:57:15.234441 6143 raft_consensus.cc:3054] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.176932 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet d1afe41a09954fc8873ea53a330ea03f (table default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c [id=0f2abb2b06c2485aab55a063ad4a0b47]) successfully deleted
W20250110 20:57:15.237219 4887 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 60299.5us
I20250110 20:57:15.237677 6129 leader_election.cc:290] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:15.160210 4898 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 49) took 2150 ms (client timeout 1000 ms). Trace:
W20250110 20:57:15.112406 4899 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 57) took 1835 ms (client timeout 1000 ms). Trace:
I20250110 20:57:15.169677 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c951d5ca47e8409db646b43bb04a7034" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:15.245297 5110 leader_election.cc:304] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:15.246048 6059 raft_consensus.cc:2743] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:15.246703 5594 tablet_bootstrap.cc:654] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:15.243749 4899 rpcz_store.cc:269] 0110 20:57:13.276572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:13.377787 (+101215us) service_pool.cc:225] Handling call
0110 20:57:13.383486 (+ 5699us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:15.112337 (+1728851us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:14.547603 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:14.548129 (+ 526us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:14.548152 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:14.548163 (+ 11us) tablet.cc:659] Decoding operations
0110 20:57:14.548532 (+ 369us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:14.548591 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:14.548613 (+ 22us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:14.548812 (+ 199us) tablet.cc:698] Row locks acquired
0110 20:57:14.548824 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:14.548994 (+ 170us) write_op.cc:270] Start()
0110 20:57:14.549318 (+ 324us) write_op.cc:276] Timestamp: P: 1736542634548868 usec, L: 0
0110 20:57:14.549334 (+ 16us) op_driver.cc:348] REPLICATION: starting
0110 20:57:14.550038 (+ 704us) log.cc:844] Serialized 533 byte log entry
0110 20:57:14.629169 (+ 79131us) op_driver.cc:464] REPLICATION: finished
0110 20:57:14.631099 (+ 1930us) write_op.cc:301] APPLY: starting
0110 20:57:14.631153 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:14.631238 (+ 85us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:14.631323 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:14.631470 (+ 147us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:14.631589 (+ 119us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:14.631630 (+ 41us) write_op.cc:312] APPLY: finished
0110 20:57:14.631787 (+ 157us) log.cc:844] Serialized 23 byte log entry
0110 20:57:14.632404 (+ 617us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.632471 (+ 67us) write_op.cc:454] Released schema lock
0110 20:57:14.632502 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:14.632709 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.632737 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2665,"reported_tablets":10,"spinlock_wait_cycles":7153408,"child_traces":[["op",{"apply.queue_time_us":1769,"apply.run_cpu_time_us":1120,"apply.run_wall_time_us":1586,"num_ops":1,"prepare.queue_time_us":117,"prepare.run_cpu_time_us":2119,"prepare.run_wall_time_us":75358,"raft.queue_time_us":82,"raft.run_cpu_time_us":641,"raft.run_wall_time_us":835,"replication_time_us":79743}]]}
I20250110 20:57:15.248997 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_7d46d595a1834776a8052c7e8c75ada2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:15.243134 6143 raft_consensus.cc:513] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:15.250582 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d46d595a1834776a8052c7e8c75ada2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:15.252936 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1afe41a09954fc8873ea53a330ea03f" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:15.253518 5323 raft_consensus.cc:3054] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.259856 5323 raft_consensus.cc:2463] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:15.260713 5398 leader_election.cc:304] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
W20250110 20:57:15.258981 4898 rpcz_store.cc:269] 0110 20:57:13.010279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:13.010382 (+ 103us) service_pool.cc:225] Handling call
0110 20:57:13.064617 (+ 54235us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:15.160147 (+2095530us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:13.673249 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:13.700867 (+ 27618us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:13.700892 (+ 25us) write_op.cc:435] Acquired schema lock
0110 20:57:13.700906 (+ 14us) tablet.cc:659] Decoding operations
0110 20:57:13.701216 (+ 310us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:13.701256 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:13.701278 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:13.709708 (+ 8430us) tablet.cc:698] Row locks acquired
0110 20:57:13.709724 (+ 16us) write_op.cc:260] PREPARE: finished
0110 20:57:13.709834 (+ 110us) write_op.cc:270] Start()
0110 20:57:13.709914 (+ 80us) write_op.cc:276] Timestamp: P: 1736542633709774 usec, L: 0
0110 20:57:13.709930 (+ 16us) op_driver.cc:348] REPLICATION: starting
0110 20:57:13.710394 (+ 464us) log.cc:844] Serialized 913 byte log entry
0110 20:57:13.740965 (+ 30571us) op_driver.cc:464] REPLICATION: finished
0110 20:57:13.741601 (+ 636us) write_op.cc:301] APPLY: starting
0110 20:57:13.741653 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:13.741756 (+ 103us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:13.741846 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:13.742087 (+ 241us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:13.742222 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:13.742264 (+ 42us) write_op.cc:312] APPLY: finished
0110 20:57:13.742430 (+ 166us) log.cc:844] Serialized 29 byte log entry
0110 20:57:13.742565 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:13.742620 (+ 55us) write_op.cc:454] Released schema lock
0110 20:57:13.742649 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:13.742833 (+ 184us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:13.742854 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":17,"spinlock_wait_cycles":118085504,"child_traces":[["op",{"apply.queue_time_us":505,"apply.run_cpu_time_us":1202,"apply.run_wall_time_us":1202,"num_ops":2,"prepare.queue_time_us":265612,"prepare.run_cpu_time_us":2126,"prepare.run_wall_time_us":39662,"raft.queue_time_us":74,"raft.run_cpu_time_us":508,"raft.run_wall_time_us":507,"replication_time_us":30948}]]}
I20250110 20:57:15.261569 6129 raft_consensus.cc:2798] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:15.262099 6129 raft_consensus.cc:695] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:15.264461 6129 consensus_queue.cc:237] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.212515 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44ca47c747544b6afec83b399dcdcb5" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:15.208603 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet ae5956b1aa17449497f9695a111ebb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 4b28befa048244309a06f69b8a3b4b8f) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.270498 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.170137 5437 tablet_service.cc:1514] Processing DeleteTablet for tablet 31b65488c884452d876ced1c3d1efb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:15.275420 5255 leader_election.cc:343] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.276883 6130 consensus_queue.cc:1035] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:15.212668 5592 log.cc:1198] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/19d6eca349514b9fb0fff7cd6230027c
I20250110 20:57:15.278649 5592 ts_tablet_manager.cc:1939] T 19d6eca349514b9fb0fff7cd6230027c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:15.170902 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.279330 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 166) took 2495 ms. Trace:
I20250110 20:57:15.279587 5592 rpcz_store.cc:276] 0110 20:57:12.783479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.783597 (+ 118us) service_pool.cc:225] Handling call
0110 20:57:14.808737 (+2025140us) ts_tablet_manager.cc:1145] Deleting tablet 19d6eca349514b9fb0fff7cd6230027c
0110 20:57:14.808773 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:14.963538 (+154765us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:14.996013 (+ 32475us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:15.279279 (+283266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2024653}
I20250110 20:57:15.280706 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 167) took 2359 ms. Trace:
I20250110 20:57:15.280959 5592 rpcz_store.cc:276] 0110 20:57:12.921467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:12.921595 (+ 128us) service_pool.cc:225] Handling call
0110 20:57:15.280421 (+2358826us) ts_tablet_manager.cc:1145] Deleting tablet f3104928188c489d93bfd499d148fa5d
0110 20:57:15.280459 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:15.280664 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280961}
I20250110 20:57:15.285368 5592 tablet_replica.cc:331] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:15.286162 5592 raft_consensus.cc:2238] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:15.287194 5592 pending_rounds.cc:62] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:15.177345 6132 raft_consensus.cc:491] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.288551 6132 raft_consensus.cc:513] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:15.292524 5111 leader_election.cc:343] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:15.294513 6132 leader_election.cc:290] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:15.297935 5172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44ca47c747544b6afec83b399dcdcb5" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:15.302948 5773 tablet_replica.cc:331] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:15.303648 5773 raft_consensus.cc:2238] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:15.304412 5773 raft_consensus.cc:2267] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:15.306250 5597 tablet_bootstrap.cc:492] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:15.311132 5597 tablet_bootstrap.cc:654] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.314985 5773 ts_tablet_manager.cc:1905] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.287142 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 19d6eca349514b9fb0fff7cd6230027c (table default.Table_931826e8e40a4d04a1382a2816608f99 [id=bb54800eb7624f4482ddee01f59494ff]) successfully deleted
I20250110 20:57:15.318406 4898 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ac07ac6fc9f477ba7f9accc5533be39 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:15.319828 4898 master_service.cc:946] ReplaceTablet: received request to replace tablet 7462d21b91c2441fb9e11383466e9ca7 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:15.324097 5597 tablet_bootstrap.cc:492] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:15.324538 5597 ts_tablet_manager.cc:1397] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20250110 20:57:15.326292 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 32cf221df30e4e78bd2a8078c26d29ba (DEFAULT_TABLE table=default.Table_3f8bafa0c19642adb6062c8156965b27 [id=f0fdb4ee34df445b9b8cf660d3ae239d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.326671 5597 raft_consensus.cc:357] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.327699 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 5efe92b10aea46608d55d5e5ac36eca1 (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.329051 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cf221df30e4e78bd2a8078c26d29ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.332983 5597 raft_consensus.cc:383] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.333284 5597 raft_consensus.cc:738] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.333606 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efe92b10aea46608d55d5e5ac36eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.333829 5597 consensus_queue.cc:260] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.086905 4891 rpcz_store.cc:276] 0110 20:57:13.774353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.038834 (+1264481us) service_pool.cc:225] Handling call
0110 20:57:15.040989 (+ 2155us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:15.042589 (+ 1600us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:15.085716 (+ 43127us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.085743 (+ 27us) write_op.cc:454] Released schema lock
0110 20:57:15.085966 (+ 223us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:15.086261 (+ 295us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:15.086487 (+ 226us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.050199 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.050508 (+ 309us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.050530 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:15.050543 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:15.050777 (+ 234us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.050813 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.050833 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:15.051035 (+ 202us) tablet.cc:698] Row locks acquired
0110 20:57:15.051048 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:15.051149 (+ 101us) write_op.cc:270] Start()
0110 20:57:15.051228 (+ 79us) write_op.cc:276] Timestamp: P: 1736542635051059 usec, L: 0
0110 20:57:15.051243 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.051679 (+ 436us) log.cc:844] Serialized 433 byte log entry
0110 20:57:15.059960 (+ 8281us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.060252 (+ 292us) write_op.cc:301] APPLY: starting
0110 20:57:15.060303 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.060394 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.060481 (+ 87us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.060810 (+ 329us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.060930 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:15.060978 (+ 48us) write_op.cc:312] APPLY: finished
0110 20:57:15.061137 (+ 159us) log.cc:844] Serialized 29 byte log entry
0110 20:57:15.061262 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.061323 (+ 61us) write_op.cc:454] Released schema lock
0110 20:57:15.061351 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":157,"apply.run_cpu_time_us":1271,"apply.run_wall_time_us":1271,"num_ops":2,"prepare.queue_time_us":79,"prepare.run_cpu_time_us":1739,"prepare.run_wall_time_us":3901,"raft.queue_time_us":71,"raft.run_cpu_time_us":505,"raft.run_wall_time_us":504,"replication_time_us":8627}]]}
I20250110 20:57:15.335840 5597 ts_tablet_manager.cc:1428] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250110 20:57:15.342661 5597 tablet_bootstrap.cc:492] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:15.350118 5597 tablet_bootstrap.cc:654] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.352607 4896 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36568 (request call id 220) took 1480 ms. Trace:
I20250110 20:57:15.353717 4896 rpcz_store.cc:276] 0110 20:57:13.872189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.198732 (+1326543us) service_pool.cc:225] Handling call
0110 20:57:15.203111 (+ 4379us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:15.204450 (+ 1339us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:15.352002 (+147552us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:15.352300 (+ 298us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:15.352548 (+ 248us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.218741 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.219083 (+ 342us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.219106 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:15.219120 (+ 14us) tablet.cc:659] Decoding operations
0110 20:57:15.219409 (+ 289us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.219446 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.219468 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:15.219655 (+ 187us) tablet.cc:698] Row locks acquired
0110 20:57:15.219669 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:15.219775 (+ 106us) write_op.cc:270] Start()
0110 20:57:15.219854 (+ 79us) write_op.cc:276] Timestamp: P: 1736542635219725 usec, L: 0
0110 20:57:15.219868 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.220540 (+ 672us) log.cc:844] Serialized 433 byte log entry
0110 20:57:15.347643 (+127103us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.348231 (+ 588us) write_op.cc:301] APPLY: starting
0110 20:57:15.348280 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.348375 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.348466 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.348824 (+ 358us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.348973 (+ 149us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:15.349013 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:15.349178 (+ 165us) log.cc:844] Serialized 29 byte log entry
0110 20:57:15.349310 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.349370 (+ 60us) write_op.cc:454] Released schema lock
0110 20:57:15.349399 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:15.349635 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.349658 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":448,"apply.run_cpu_time_us":1376,"apply.run_wall_time_us":1377,"num_ops":2,"prepare.queue_time_us":108,"prepare.run_cpu_time_us":1808,"prepare.run_wall_time_us":2957,"replication_time_us":127682}]]}
I20250110 20:57:15.359040 4894 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36568 (request call id 221) took 1359 ms. Trace:
I20250110 20:57:15.361800 4894 rpcz_store.cc:276] 0110 20:57:13.999732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.208557 (+1208825us) service_pool.cc:225] Handling call
0110 20:57:15.212535 (+ 3978us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:15.222642 (+ 10107us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:15.358423 (+135781us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:15.358725 (+ 302us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:15.358982 (+ 257us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.223809 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.224141 (+ 332us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.224163 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:15.224186 (+ 23us) tablet.cc:659] Decoding operations
0110 20:57:15.224658 (+ 472us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.224693 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.224716 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:15.224891 (+ 175us) tablet.cc:698] Row locks acquired
0110 20:57:15.224905 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:15.225005 (+ 100us) write_op.cc:270] Start()
0110 20:57:15.225082 (+ 77us) write_op.cc:276] Timestamp: P: 1736542635224958 usec, L: 0
0110 20:57:15.225096 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.225572 (+ 476us) log.cc:844] Serialized 433 byte log entry
0110 20:57:15.347859 (+122287us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.349855 (+ 1996us) write_op.cc:301] APPLY: starting
0110 20:57:15.349899 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.350000 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.350093 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.350427 (+ 334us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.350547 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:15.350584 (+ 37us) write_op.cc:312] APPLY: finished
0110 20:57:15.350731 (+ 147us) log.cc:844] Serialized 29 byte log entry
0110 20:57:15.350850 (+ 119us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.350912 (+ 62us) write_op.cc:454] Released schema lock
0110 20:57:15.350969 (+ 57us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1891,"apply.run_cpu_time_us":1278,"apply.run_wall_time_us":1277,"num_ops":2,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":1750,"prepare.run_wall_time_us":2837,"replication_time_us":122691,"threads_started":1}]]}
I20250110 20:57:15.287647 5592 pending_rounds.cc:69] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878633393537024 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "d310cf52aa8c4930a7821117a46dc958" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_5053ea7526a74f6da2ece254f837bbae" dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" new_extra_config { } }
W20250110 20:57:15.376245 5592 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40232: Aborted: Op aborted
I20250110 20:57:15.377105 5592 raft_consensus.cc:2267] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:15.279640 6143 leader_election.cc:290] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:15.354903 5773 ts_tablet_manager.cc:1918] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:15.383615 5773 log.cc:1198] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/e27e43461251413b97721323cd935374
I20250110 20:57:15.384326 5773 ts_tablet_manager.cc:1939] T e27e43461251413b97721323cd935374 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:15.393926 5773 tablet_replica.cc:331] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:15.394603 5773 raft_consensus.cc:2238] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:15.395032 5773 raft_consensus.cc:2267] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:15.395073 5597 tablet_bootstrap.cc:492] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:15.395478 5597 ts_tablet_manager.cc:1397] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.000s
I20250110 20:57:15.396659 5773 ts_tablet_manager.cc:1905] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.397600 5597 raft_consensus.cc:357] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.398422 5597 raft_consensus.cc:383] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.398664 5597 raft_consensus.cc:738] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.399382 5597 consensus_queue.cc:260] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.402794 5597 ts_tablet_manager.cc:1428] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20250110 20:57:15.402820 4885 catalog_manager.cc:4951] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): alter failed for tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]): Aborted: Op aborted
I20250110 20:57:15.403538 5597 tablet_bootstrap.cc:492] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:15.403512 5594 tablet_bootstrap.cc:492] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:15.403879 5594 ts_tablet_manager.cc:1397] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.393s user 0.009s sys 0.000s
I20250110 20:57:15.406366 6047 raft_consensus.cc:491] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.406759 6047 raft_consensus.cc:513] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.407483 5592 ts_tablet_manager.cc:1905] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.408977 6047 leader_election.cc:290] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:15.410542 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7f09df8e274f3e9542292d5c1ec6c6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:15.411103 5459 raft_consensus.cc:2463] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:15.412005 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 5efe92b10aea46608d55d5e5ac36eca1 (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:15.413385 5398 consensus_peers.cc:487] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c951d5ca47e8409db646b43bb04a7034. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:15.414176 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 32cf221df30e4e78bd2a8078c26d29ba (DEFAULT_TABLE table=default.Table_3f8bafa0c19642adb6062c8156965b27 [id=f0fdb4ee34df445b9b8cf660d3ae239d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.414064 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d67e94c8f9a414499b5e20a5cc0ead3" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:15.415299 5458 raft_consensus.cc:2463] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:15.416668 5255 leader_election.cc:304] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:15.419456 6047 raft_consensus.cc:2798] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:15.419761 6047 raft_consensus.cc:491] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:15.420032 6047 raft_consensus.cc:3054] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.438340 5594 raft_consensus.cc:357] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.439288 5594 raft_consensus.cc:383] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.439587 5594 raft_consensus.cc:738] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.440291 5594 consensus_queue.cc:260] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.442317 5594 ts_tablet_manager.cc:1428] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.038s user 0.004s sys 0.002s
I20250110 20:57:15.443567 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efe92b10aea46608d55d5e5ac36eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.458737 5600 tablet_bootstrap.cc:492] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:15.459117 5600 ts_tablet_manager.cc:1397] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.329s user 0.038s sys 0.060s
I20250110 20:57:15.459605 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cf221df30e4e78bd2a8078c26d29ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.461946 5594 tablet_bootstrap.cc:492] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:15.463295 6047 raft_consensus.cc:513] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.464012 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 4b28befa048244309a06f69b8a3b4b8f (DEFAULT_TABLE table=default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.465010 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b28befa048244309a06f69b8a3b4b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.467470 6130 raft_consensus.cc:491] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.467849 6130 raft_consensus.cc:513] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.469760 6130 leader_election.cc:290] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:15.470789 6047 leader_election.cc:290] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:15.479521 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:15.482131 5458 raft_consensus.cc:2463] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:15.482952 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7f09df8e274f3e9542292d5c1ec6c6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:15.483836 5255 leader_election.cc:304] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:15.484807 6047 raft_consensus.cc:2798] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:15.485177 6047 raft_consensus.cc:491] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:15.485556 6047 raft_consensus.cc:3054] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.477432 5600 raft_consensus.cc:357] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.488350 5110 leader_election.cc:304] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:15.483465 5462 raft_consensus.cc:3054] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.494777 5462 raft_consensus.cc:2463] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:15.501731 6172 raft_consensus.cc:2798] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:15.502240 6172 raft_consensus.cc:491] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:15.502568 6172 raft_consensus.cc:3054] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.505002 5594 tablet_bootstrap.cc:654] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.506309 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a0178b82e1d8457eb7f165640e222c72" } modify_external_catalogs: true
I20250110 20:57:15.506856 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a0178b82e1d8457eb7f165640e222c72" } modify_external_catalogs: true
W20250110 20:57:15.489765 5257 leader_election.cc:343] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.511801 5257 leader_election.cc:304] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:15.512982 6130 raft_consensus.cc:2743] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:15.513710 5592 ts_tablet_manager.cc:1918] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:15.514039 5592 log.cc:1198] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/d310cf52aa8c4930a7821117a46dc958
I20250110 20:57:15.495723 5255 leader_election.cc:304] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:15.514672 5592 ts_tablet_manager.cc:1939] T d310cf52aa8c4930a7821117a46dc958 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:15.516410 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 168) took 1680 ms. Trace:
I20250110 20:57:15.488999 5600 raft_consensus.cc:383] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.517006 5600 raft_consensus.cc:738] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.516675 5592 rpcz_store.cc:276] 0110 20:57:13.835871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:13.836020 (+ 149us) service_pool.cc:225] Handling call
0110 20:57:15.285242 (+1449222us) ts_tablet_manager.cc:1145] Deleting tablet d310cf52aa8c4930a7821117a46dc958
0110 20:57:15.285276 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:15.464869 (+179593us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:15.513514 (+ 48645us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:15.516366 (+ 2852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448637}
I20250110 20:57:15.517591 5600 consensus_queue.cc:260] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:15.522097 5399 leader_election.cc:343] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: d1afe41a09954fc8873ea53a330ea03f
I20250110 20:57:15.523883 6172 raft_consensus.cc:513] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.526050 6172 leader_election.cc:290] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:15.527876 6176 raft_consensus.cc:2798] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:15.530824 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d67e94c8f9a414499b5e20a5cc0ead3" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:15.531405 5463 raft_consensus.cc:3054] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.537999 5600 ts_tablet_manager.cc:1428] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.079s user 0.008s sys 0.000s
I20250110 20:57:15.538841 5600 tablet_bootstrap.cc:492] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:15.540766 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 4b28befa048244309a06f69b8a3b4b8f (DEFAULT_TABLE table=default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.541714 6047 raft_consensus.cc:513] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.542341 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 5efe92b10aea46608d55d5e5ac36eca1 (DEFAULT_TABLE table=default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.543612 5171 raft_consensus.cc:3054] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.544271 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 32cf221df30e4e78bd2a8078c26d29ba (DEFAULT_TABLE table=default.Table_3f8bafa0c19642adb6062c8156965b27 [id=f0fdb4ee34df445b9b8cf660d3ae239d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.547724 6176 raft_consensus.cc:695] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Becoming Leader. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Running, Role: LEADER
I20250110 20:57:15.541383 5463 raft_consensus.cc:2463] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:15.549631 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:15.549294 6176 consensus_queue.cc:237] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.551086 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7f09df8e274f3e9542292d5c1ec6c6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:15.551913 5174 raft_consensus.cc:2463] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:15.543601 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b28befa048244309a06f69b8a3b4b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.554432 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32cf221df30e4e78bd2a8078c26d29ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.554461 4894 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_41125035ac2544fb9a27c3c25133b711" } modify_external_catalogs: true
I20250110 20:57:15.555050 4894 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_41125035ac2544fb9a27c3c25133b711" } modify_external_catalogs: true
I20250110 20:57:15.557744 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efe92b10aea46608d55d5e5ac36eca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.570000 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb7f09df8e274f3e9542292d5c1ec6c6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:15.570582 5175 raft_consensus.cc:3054] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.574244 5176 raft_consensus.cc:3054] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.574852 5597 tablet_bootstrap.cc:654] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.553242 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 169) took 1082 ms. Trace:
I20250110 20:57:15.577251 5592 rpcz_store.cc:276] 0110 20:57:14.470435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.470558 (+ 123us) service_pool.cc:225] Handling call
0110 20:57:15.548739 (+1078181us) ts_tablet_manager.cc:1145] Deleting tablet 48e53d95ef8645c0bd123138655305ac
0110 20:57:15.549055 (+ 316us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:15.552699 (+ 3644us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1076961}
I20250110 20:57:15.582712 6047 leader_election.cc:290] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:15.589102 5592 tablet_replica.cc:331] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:15.593874 5592 raft_consensus.cc:2238] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:15.594470 5592 pending_rounds.cc:62] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:15.594681 5592 pending_rounds.cc:69] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878634049576960 op_type: NO_OP noop_request { }
I20250110 20:57:15.594997 5592 raft_consensus.cc:2883] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:15.595203 5600 tablet_bootstrap.cc:654] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.595297 5592 raft_consensus.cc:2267] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:15.597842 6169 log.cc:926] Time spent T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Append to log took a long time: real 0.100s user 0.000s sys 0.000s
I20250110 20:57:15.599483 5592 ts_tablet_manager.cc:1905] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.604861 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:15.605389 5460 raft_consensus.cc:3054] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.611785 5460 raft_consensus.cc:2463] T bd9d0249798e4f24adf0b36d9e059b7e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:15.517849 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]) successfully deleted
W20250110 20:57:15.618068 5257 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 105387us
W20250110 20:57:15.618943 4885 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 101100us
I20250110 20:57:15.619084 5594 tablet_bootstrap.cc:492] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:15.619431 5594 ts_tablet_manager.cc:1397] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.158s user 0.013s sys 0.001s
W20250110 20:57:15.620570 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 48e53d95ef8645c0bd123138655305ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e53d95ef8645c0bd123138655305ac already in progress: creating tablet
I20250110 20:57:15.621446 5594 raft_consensus.cc:357] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.622026 5594 raft_consensus.cc:383] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.622272 5594 raft_consensus.cc:738] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.622505 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 4b28befa048244309a06f69b8a3b4b8f (DEFAULT_TABLE table=default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.622859 5594 consensus_queue.cc:260] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.623535 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b28befa048244309a06f69b8a3b4b8f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:15.624560 5257 leader_election.cc:343] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.625638 5293 tablet_service.cc:1467] Processing CreateTablet for tablet bb535174548d496e920801723e0909b3 (DEFAULT_TABLE table=default.Table_57c42d638822484caa513bd1ce8616d9 [id=83c23c863af4460296277c44bddc35fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.626616 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb535174548d496e920801723e0909b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.632390 5255 leader_election.cc:304] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:15.633067 6130 raft_consensus.cc:2798] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:15.633500 6130 raft_consensus.cc:695] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Becoming Leader. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Running, Role: LEADER
I20250110 20:57:15.634219 6130 consensus_queue.cc:237] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.635592 5594 ts_tablet_manager.cc:1428] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20250110 20:57:15.636370 5594 tablet_bootstrap.cc:492] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:15.641069 5594 tablet_bootstrap.cc:654] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.641368 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6eca349514b9fb0fff7cd6230027c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.645251 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1ca1670916401ab0a43685df1b6759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.647028 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6eca349514b9fb0fff7cd6230027c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.649757 6177 consensus_queue.cc:1035] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:15.650439 5438 tablet_service.cc:1467] Processing CreateTablet for tablet bb535174548d496e920801723e0909b3 (DEFAULT_TABLE table=default.Table_57c42d638822484caa513bd1ce8616d9 [id=83c23c863af4460296277c44bddc35fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.651413 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb535174548d496e920801723e0909b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:15.654219 5111 leader_election.cc:343] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3
W20250110 20:57:15.666417 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 48546.7us
I20250110 20:57:15.667526 5110 leader_election.cc:304] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:15.668222 6172 consensus_queue.cc:1035] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:15.669355 6172 raft_consensus.cc:2798] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:15.669350 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:15.669817 6172 raft_consensus.cc:695] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
W20250110 20:57:15.673012 5257 leader_election.cc:343] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.673550 6172 consensus_queue.cc:237] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:15.681131 4887 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.252:37233, user_credentials={real_user=slave}} blocked reactor thread for 246769us
I20250110 20:57:15.688311 5594 tablet_bootstrap.cc:492] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:15.688730 5594 ts_tablet_manager.cc:1397] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.000s
I20250110 20:57:15.691004 5594 raft_consensus.cc:357] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.691620 5594 raft_consensus.cc:383] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.691855 5594 raft_consensus.cc:738] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.692389 5594 consensus_queue.cc:260] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.694679 5594 ts_tablet_manager.cc:1428] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250110 20:57:15.688804 5592 ts_tablet_manager.cc:1918] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:15.695446 5594 tablet_bootstrap.cc:492] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:15.695595 5592 log.cc:1198] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/d1afe41a09954fc8873ea53a330ea03f
I20250110 20:57:15.696882 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]) successfully deleted
I20250110 20:57:15.702737 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 79d990d874d34f98ade9b074496d5070 (DEFAULT_TABLE table=default.Table_cf6aeef756774db6a25af02ae4a44197 [id=c4114f1d8e0b46b18b2c2cbd3d2430df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.703538 5592 ts_tablet_manager.cc:1939] T d1afe41a09954fc8873ea53a330ea03f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:15.703933 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79d990d874d34f98ade9b074496d5070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.709703 5594 tablet_bootstrap.cc:654] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.715093 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 527bbddda8184a6eb67c2411ae50db63 with delete_type TABLET_DATA_DELETED (replaced by tablet 9633eb4580c04f11a48ff5055e257c50) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.716339 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 172) took 1112 ms. Trace:
I20250110 20:57:15.716650 5592 rpcz_store.cc:276] 0110 20:57:14.596678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.596817 (+ 139us) service_pool.cc:225] Handling call
0110 20:57:15.588965 (+992148us) ts_tablet_manager.cc:1145] Deleting tablet d1afe41a09954fc8873ea53a330ea03f
0110 20:57:15.589004 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:15.612955 (+ 23951us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:15.688596 (+ 75641us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:15.708536 (+ 19940us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":991622}
I20250110 20:57:15.717633 5592 tablet_replica.cc:331] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:15.718350 5592 raft_consensus.cc:2238] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:15.717772 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet d1afe41a09954fc8873ea53a330ea03f (table default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c [id=0f2abb2b06c2485aab55a063ad4a0b47]) successfully deleted
I20250110 20:57:15.718773 5592 raft_consensus.cc:2267] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:15.719390 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 2d634add82994f74960cd2b8373a4dc6 because the tablet was not found. No further retry: Not found: Tablet not found: 2d634add82994f74960cd2b8373a4dc6
I20250110 20:57:15.720361 5592 ts_tablet_manager.cc:1905] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.721443 6130 raft_consensus.cc:491] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.721848 6130 raft_consensus.cc:513] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.723572 6130 leader_election.cc:290] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:15.717649 5594 tablet_bootstrap.cc:492] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:15.733565 5594 ts_tablet_manager.cc:1397] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.000s
W20250110 20:57:15.740582 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 961fef96e9124aafb1c51e287f603b64 because the tablet was not found. No further retry: Not found: Tablet not found: 961fef96e9124aafb1c51e287f603b64
I20250110 20:57:15.740568 6192 tablet.cc:1721] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 20:57:15.733752 4897 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 40) took 1545 ms (client timeout 1000 ms). Trace:
I20250110 20:57:15.749889 5594 raft_consensus.cc:357] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.750638 5594 raft_consensus.cc:383] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:15.750921 5594 raft_consensus.cc:738] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:15.751495 5594 consensus_queue.cc:260] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.753932 5594 ts_tablet_manager.cc:1428] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250110 20:57:15.754794 5594 tablet_bootstrap.cc:492] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:15.759663 5594 tablet_bootstrap.cc:654] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:15.768929 5148 tablet_service.cc:1467] Processing CreateTablet for tablet bb535174548d496e920801723e0909b3 (DEFAULT_TABLE table=default.Table_57c42d638822484caa513bd1ce8616d9 [id=83c23c863af4460296277c44bddc35fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.769784 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.770668 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:15.776247 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc7b1acf4114bf69d801437855760df with delete_type TABLET_DATA_DELETED (replaced by tablet 5efe92b10aea46608d55d5e5ac36eca1) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.795647 5595 ts_tablet_manager.cc:1918] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:15.814821 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:15.818251 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb535174548d496e920801723e0909b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:15.824631 4900 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 50) took 1796 ms (client timeout 1000 ms). Trace:
I20250110 20:57:15.760844 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet cd0a666892d04e97a4b9967e38ab0946 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.774266 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet d310cf52aa8c4930a7821117a46dc958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4aaa42bc5ab042e79baee1ea838851eb) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.775540 5592 ts_tablet_manager.cc:1918] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:15.782663 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 79d990d874d34f98ade9b074496d5070 (DEFAULT_TABLE table=default.Table_cf6aeef756774db6a25af02ae4a44197 [id=c4114f1d8e0b46b18b2c2cbd3d2430df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.784224 5437 tablet_service.cc:1514] Processing DeleteTablet for tablet c951d5ca47e8409db646b43bb04a7034 with delete_type TABLET_DATA_DELETED (replaced by tablet 95b52e115a8e4bae95da0ef9d5afff8c) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.803805 6195 tablet.cc:1721] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:15.829145 5592 log.cc:1198] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/e27e43461251413b97721323cd935374
I20250110 20:57:15.829840 5592 ts_tablet_manager.cc:1939] T e27e43461251413b97721323cd935374 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:15.830693 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79d990d874d34f98ade9b074496d5070. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:15.832201 5592 tablet_replica.cc:331] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:15.816928 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:15.832903 5592 raft_consensus.cc:2238] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:15.834831 5592 pending_rounds.cc:62] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:15.835057 5592 pending_rounds.cc:69] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878623476854784 op_type: NO_OP noop_request { }
I20250110 20:57:15.835381 5592 raft_consensus.cc:2883] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:15.835659 5592 raft_consensus.cc:2267] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:15.837250 5592 ts_tablet_manager.cc:1905] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.840550 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 79d990d874d34f98ade9b074496d5070 (DEFAULT_TABLE table=default.Table_cf6aeef756774db6a25af02ae4a44197 [id=c4114f1d8e0b46b18b2c2cbd3d2430df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:15.841576 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79d990d874d34f98ade9b074496d5070. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:15.826226 4893 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 58) took 1498 ms (client timeout 1000 ms). Trace:
I20250110 20:57:15.863646 5592 ts_tablet_manager.cc:1918] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:15.863986 5592 log.cc:1198] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:15.864629 5592 ts_tablet_manager.cc:1939] T 3a520230d9fd4a74914e46f784dcf747 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
W20250110 20:57:15.862093 4900 rpcz_store.cc:269] 0110 20:57:14.028598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.045986 (+ 17388us) service_pool.cc:225] Handling call
0110 20:57:15.824568 (+1778582us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:14.898146 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:14.898516 (+ 370us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:14.898542 (+ 26us) write_op.cc:435] Acquired schema lock
0110 20:57:14.898555 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:14.898864 (+ 309us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:14.898903 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:14.898925 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:14.899170 (+ 245us) tablet.cc:698] Row locks acquired
0110 20:57:14.899183 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:14.899290 (+ 107us) write_op.cc:270] Start()
0110 20:57:14.899367 (+ 77us) write_op.cc:276] Timestamp: P: 1736542634899218 usec, L: 0
0110 20:57:14.899380 (+ 13us) op_driver.cc:348] REPLICATION: starting
0110 20:57:14.899826 (+ 446us) log.cc:844] Serialized 913 byte log entry
0110 20:57:14.922976 (+ 23150us) op_driver.cc:464] REPLICATION: finished
0110 20:57:14.923563 (+ 587us) write_op.cc:301] APPLY: starting
0110 20:57:14.923624 (+ 61us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:14.923750 (+ 126us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:14.923856 (+ 106us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:14.924092 (+ 236us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:14.924218 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:14.924257 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:57:14.924437 (+ 180us) log.cc:844] Serialized 29 byte log entry
0110 20:57:14.924571 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.924636 (+ 65us) write_op.cc:454] Released schema lock
0110 20:57:14.924665 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:14.924872 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.924894 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":33075,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":408,"apply.run_cpu_time_us":1282,"apply.run_wall_time_us":1283,"num_ops":2,"prepare.queue_time_us":61137,"prepare.run_cpu_time_us":1962,"prepare.run_wall_time_us":3679,"raft.queue_time_us":87,"raft.run_cpu_time_us":530,"raft.run_wall_time_us":530,"replication_time_us":23290}]]}
I20250110 20:57:15.867609 5175 raft_consensus.cc:2463] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:15.871124 6144 raft_consensus.cc:491] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.871866 6144 raft_consensus.cc:513] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:15.744099 4897 rpcz_store.cc:269] 0110 20:57:14.188215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.479926 (+291711us) service_pool.cc:225] Handling call
0110 20:57:15.090011 (+610085us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.090037 (+ 26us) write_op.cc:454] Released schema lock
0110 20:57:15.733680 (+643643us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:14.974001 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:14.974345 (+ 344us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:14.974367 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:14.974379 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:14.974619 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:14.974653 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:14.974694 (+ 41us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:14.974821 (+ 127us) tablet.cc:698] Row locks acquired
0110 20:57:14.974833 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:14.974934 (+ 101us) write_op.cc:270] Start()
0110 20:57:14.975007 (+ 73us) write_op.cc:276] Timestamp: P: 1736542634974853 usec, L: 0
0110 20:57:14.975022 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:14.981533 (+ 6511us) log.cc:844] Serialized 356 byte log entry
0110 20:57:14.989843 (+ 8310us) op_driver.cc:464] REPLICATION: finished
0110 20:57:14.990695 (+ 852us) write_op.cc:301] APPLY: starting
0110 20:57:14.990751 (+ 56us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:14.990833 (+ 82us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:14.990918 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:14.991092 (+ 174us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:14.991225 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:14.991261 (+ 36us) write_op.cc:312] APPLY: finished
0110 20:57:14.991412 (+ 151us) log.cc:844] Serialized 23 byte log entry
0110 20:57:14.991546 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:14.991609 (+ 63us) write_op.cc:454] Released schema lock
0110 20:57:14.991638 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":216539,"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":714,"apply.run_cpu_time_us":1100,"apply.run_wall_time_us":1099,"num_ops":1,"prepare.queue_time_us":88,"prepare.run_cpu_time_us":1810,"prepare.run_wall_time_us":9452,"raft.queue_time_us":76,"raft.run_cpu_time_us":569,"raft.run_wall_time_us":569,"replication_time_us":14734}]]}
I20250110 20:57:15.833609 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]) successfully deleted
I20250110 20:57:15.878248 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 3a520230d9fd4a74914e46f784dcf747 (table default.Table_8df7bbf6722249b2be68514a607459b9 [id=64cfe285ff2041778f15cf7e6020e9cb]) successfully deleted
W20250110 20:57:15.835736 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fdb2ee9419244a9ab939be56b2d7c72 consensus state has no leader
W20250110 20:57:15.836488 4892 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7961f8bd3ff42ffab214a04c704fd26 has no consensus state
I20250110 20:57:15.849103 4899 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36568 (request call id 224) took 1601 ms. Trace:
W20250110 20:57:15.816181 5257 leader_election.cc:343] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.881242 5595 log.cc:1198] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/d310cf52aa8c4930a7821117a46dc958
W20250110 20:57:15.881511 5257 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 65326.1us
I20250110 20:57:15.882930 5595 ts_tablet_manager.cc:1939] T d310cf52aa8c4930a7821117a46dc958 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:15.875981 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:15.885063 5323 raft_consensus.cc:2463] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:15.854478 4898 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 7462d21b91c2441fb9e11383466e9ca7
W20250110 20:57:15.894860 5255 leader_election.cc:343] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.896028 5255 leader_election.cc:304] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:15.855587 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet c57941aee12a4ebea2b47f4727fc6eee
I20250110 20:57:15.876497 6144 leader_election.cc:290] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:15.862509 4893 rpcz_store.cc:269] 0110 20:57:14.327604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.727383 (+399779us) service_pool.cc:225] Handling call
0110 20:57:15.280717 (+553334us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.280741 (+ 24us) write_op.cc:454] Released schema lock
0110 20:57:15.826159 (+545418us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.191341 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.191683 (+ 342us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.191706 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:15.191718 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:15.191952 (+ 234us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.191991 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.192012 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:15.192145 (+ 133us) tablet.cc:698] Row locks acquired
0110 20:57:15.192157 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:15.192273 (+ 116us) write_op.cc:270] Start()
0110 20:57:15.192352 (+ 79us) write_op.cc:276] Timestamp: P: 1736542635192230 usec, L: 0
0110 20:57:15.192366 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.192823 (+ 457us) log.cc:844] Serialized 533 byte log entry
0110 20:57:15.262865 (+ 70042us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.263239 (+ 374us) write_op.cc:301] APPLY: starting
0110 20:57:15.263289 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.263375 (+ 86us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.263466 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.263617 (+ 151us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.263734 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:15.263771 (+ 37us) write_op.cc:312] APPLY: finished
0110 20:57:15.263934 (+ 163us) log.cc:844] Serialized 23 byte log entry
0110 20:57:15.264078 (+ 144us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.264131 (+ 53us) write_op.cc:454] Released schema lock
0110 20:57:15.264166 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":249,"apply.run_cpu_time_us":1073,"apply.run_wall_time_us":1072,"num_ops":1,"prepare.queue_time_us":102,"prepare.run_cpu_time_us":1738,"prepare.run_wall_time_us":3335,"raft.queue_time_us":72,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":503,"replication_time_us":70411}]]}
I20250110 20:57:15.888631 5398 leader_election.cc:304] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:15.908855 6133 raft_consensus.cc:2798] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:15.909159 6133 raft_consensus.cc:491] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:15.909480 6133 raft_consensus.cc:3054] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.910454 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462d21b91c2441fb9e11383466e9ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f8713c1e5a743a489de11bc65857bff) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.894255 6145 raft_consensus.cc:491] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:15.914676 6145 raft_consensus.cc:513] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.916424 6145 leader_election.cc:290] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:15.918754 5398 leader_election.cc:343] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 491626848ad54be3a18186c65d6dbb60
I20250110 20:57:15.920424 6133 raft_consensus.cc:513] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:15.927848 6130 raft_consensus.cc:2743] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:15.930387 5294 tablet_service.cc:1514] Processing DeleteTablet for tablet c57941aee12a4ebea2b47f4727fc6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.932055 6144 consensus_queue.cc:1035] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:15.934607 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462d21b91c2441fb9e11383466e9ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f8713c1e5a743a489de11bc65857bff) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.935864 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet c57941aee12a4ebea2b47f4727fc6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.936657 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 165) took 1828 ms. Trace:
I20250110 20:57:15.856616 4894 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet bae52ea080f54d228b6b76922248ed5a
I20250110 20:57:15.937633 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:15.938163 5323 raft_consensus.cc:3054] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.938845 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f21a12015eab4635890b871de037c960"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:15.937362 6133 leader_election.cc:290] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:15.943310 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet bae52ea080f54d228b6b76922248ed5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:15.946602 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:15.895514 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:15.929849 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491626848ad54be3a18186c65d6dbb60" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:15.949003 5171 raft_consensus.cc:2463] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:15.949532 4895 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36568 (request call id 223) took 1751 ms. Trace:
I20250110 20:57:15.950335 4892 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:36568 (request call id 222) took 1826 ms. Trace:
I20250110 20:57:15.936947 5595 rpcz_store.cc:276] 0110 20:57:14.108147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.108435 (+ 288us) service_pool.cc:225] Handling call
0110 20:57:15.158209 (+1049774us) ts_tablet_manager.cc:1145] Deleting tablet d310cf52aa8c4930a7821117a46dc958
0110 20:57:15.158246 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:15.499274 (+341028us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:15.795441 (+296167us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:15.936605 (+141164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1049268}
I20250110 20:57:15.949869 4895 rpcz_store.cc:276] 0110 20:57:14.198050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.233614 (+1035564us) service_pool.cc:225] Handling call
0110 20:57:15.234625 (+ 1011us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:15.235353 (+ 728us) catalog_manager.cc:3627] Apply alter schema
0110 20:57:15.235893 (+ 540us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 20:57:15.236067 (+ 174us) catalog_manager.cc:3817] Updating metadata on disk
0110 20:57:15.835266 (+599199us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 20:57:15.949477 (+114211us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.514749 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.515151 (+ 402us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.515175 (+ 24us) write_op.cc:435] Acquired schema lock
0110 20:57:15.515186 (+ 11us) tablet.cc:659] Decoding operations
0110 20:57:15.515422 (+ 236us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.515454 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.515473 (+ 19us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:15.515619 (+ 146us) tablet.cc:698] Row locks acquired
0110 20:57:15.515632 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:15.515728 (+ 96us) write_op.cc:270] Start()
0110 20:57:15.515806 (+ 78us) write_op.cc:276] Timestamp: P: 1736542635515647 usec, L: 0
0110 20:57:15.515820 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.516272 (+ 452us) log.cc:844] Serialized 405 byte log entry
0110 20:57:15.831872 (+315600us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.833382 (+ 1510us) write_op.cc:301] APPLY: starting
0110 20:57:15.833441 (+ 59us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.833528 (+ 87us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.833624 (+ 96us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.833750 (+ 126us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.833857 (+ 107us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:15.833897 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:15.834043 (+ 146us) log.cc:844] Serialized 23 byte log entry
0110 20:57:15.834152 (+ 109us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.834200 (+ 48us) write_op.cc:454] Released schema lock
0110 20:57:15.834226 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1392,"apply.run_cpu_time_us":995,"apply.run_wall_time_us":996,"mutex_wait_us":139153,"num_ops":1,"prepare.queue_time_us":277631,"prepare.run_cpu_time_us":1757,"prepare.run_wall_time_us":4378,"raft.queue_time_us":779,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":315985}]]}
I20250110 20:57:15.957296 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_1e56771d3584418e8e38ebbff845ad4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:15.962261 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e56771d3584418e8e38ebbff845ad4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:15.880340 4899 rpcz_store.cc:276] 0110 20:57:14.247268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.248561 (+1001293us) service_pool.cc:225] Handling call
0110 20:57:15.251034 (+ 2473us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:15.252794 (+ 1760us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:15.848522 (+595728us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:15.848791 (+ 269us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:15.849049 (+ 258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.519224 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.519545 (+ 321us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.519566 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:57:15.519578 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:15.519820 (+ 242us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.519851 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.519871 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:15.520054 (+ 183us) tablet.cc:698] Row locks acquired
0110 20:57:15.520067 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:15.520164 (+ 97us) write_op.cc:270] Start()
0110 20:57:15.520236 (+ 72us) write_op.cc:276] Timestamp: P: 1736542635520081 usec, L: 0
0110 20:57:15.520250 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.520653 (+ 403us) log.cc:844] Serialized 433 byte log entry
0110 20:57:15.844628 (+323975us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.845044 (+ 416us) write_op.cc:301] APPLY: starting
0110 20:57:15.845090 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.845182 (+ 92us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.845283 (+ 101us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.845712 (+ 429us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.845827 (+ 115us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:15.845867 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:15.846029 (+ 162us) log.cc:844] Serialized 29 byte log entry
0110 20:57:15.846153 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.846211 (+ 58us) write_op.cc:454] Released schema lock
0110 20:57:15.846239 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:15.846429 (+ 190us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.846453 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":285,"apply.run_cpu_time_us":1349,"apply.run_wall_time_us":1350,"mutex_wait_us":97388,"num_ops":2,"prepare.queue_time_us":240225,"prepare.run_cpu_time_us":1643,"prepare.run_wall_time_us":72221,"replication_time_us":324294}]]}
I20250110 20:57:15.956408 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]) successfully deleted
I20250110 20:57:15.929775 4896 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 227) took 1001 ms. Trace:
I20250110 20:57:15.965776 5595 tablet_replica.cc:331] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:15.966473 5595 raft_consensus.cc:2238] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:15.964241 4896 rpcz_store.cc:276] 0110 20:57:14.928451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.364938 (+436487us) service_pool.cc:225] Handling call
0110 20:57:15.506212 (+141274us) spinlock_profiling.cc:243] Waited 70.9 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b616 00007f3bfd003b44 00007f3bfd003ad2 00007f3bfd003a01 00007f3bfd00397c 00007f3bfd002c32 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:15.507349 (+ 1137us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:15.507834 (+ 485us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:15.507943 (+ 109us) catalog_manager.cc:2762] Locking tablets
0110 20:57:15.508455 (+ 512us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:15.855208 (+346753us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:15.855423 (+ 215us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:15.855516 (+ 93us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:15.929651 (+ 74135us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.591576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.591924 (+ 348us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.591946 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:15.591970 (+ 24us) tablet.cc:659] Decoding operations
0110 20:57:15.592258 (+ 288us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.592290 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.592312 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:15.592565 (+ 253us) tablet.cc:698] Row locks acquired
0110 20:57:15.592579 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:15.592679 (+ 100us) write_op.cc:270] Start()
0110 20:57:15.592753 (+ 74us) write_op.cc:276] Timestamp: P: 1736542635592587 usec, L: 0
0110 20:57:15.592768 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.593225 (+ 457us) log.cc:844] Serialized 731 byte log entry
0110 20:57:15.844835 (+251610us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.846649 (+ 1814us) write_op.cc:301] APPLY: starting
0110 20:57:15.846692 (+ 43us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.846780 (+ 88us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.846867 (+ 87us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.847067 (+ 200us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.847185 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:15.847226 (+ 41us) write_op.cc:312] APPLY: finished
0110 20:57:15.847370 (+ 144us) log.cc:844] Serialized 29 byte log entry
0110 20:57:15.847481 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.847533 (+ 52us) write_op.cc:454] Released schema lock
0110 20:57:15.847558 (+ 25us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":163004160,"child_traces":[["op",{"apply.queue_time_us":1705,"apply.run_cpu_time_us":1074,"apply.run_wall_time_us":1074,"num_ops":2,"prepare.queue_time_us":30878,"prepare.run_cpu_time_us":1839,"prepare.run_wall_time_us":2882,"replication_time_us":251998}]]}
I20250110 20:57:15.966874 5595 raft_consensus.cc:2267] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:15.930936 4898 catalog_manager.cc:6596] ReplaceTablet: tablet 7462d21b91c2441fb9e11383466e9ca7 deleted and replaced by tablet 7f8713c1e5a743a489de11bc65857bff
I20250110 20:57:15.968374 5595 ts_tablet_manager.cc:1905] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:15.968333 4898 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:36568 (request call id 226) took 1146 ms. Trace:
W20250110 20:57:15.970145 5399 leader_election.cc:343] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.970710 5399 leader_election.cc:304] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:15.968675 4898 rpcz_store.cc:276] 0110 20:57:14.821638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.319724 (+498086us) service_pool.cc:225] Handling call
0110 20:57:15.968283 (+648559us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.594560 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.594938 (+ 378us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.595056 (+ 118us) write_op.cc:435] Acquired schema lock
0110 20:57:15.595070 (+ 14us) tablet.cc:659] Decoding operations
0110 20:57:15.715789 (+120719us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.715825 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.715845 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:15.716053 (+ 208us) tablet.cc:698] Row locks acquired
0110 20:57:15.716067 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:15.716178 (+ 111us) write_op.cc:270] Start()
0110 20:57:15.716260 (+ 82us) write_op.cc:276] Timestamp: P: 1736542635716127 usec, L: 0
0110 20:57:15.716275 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.741993 (+ 25718us) log.cc:844] Serialized 662 byte log entry
0110 20:57:15.849686 (+107693us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.850148 (+ 462us) write_op.cc:301] APPLY: starting
0110 20:57:15.850194 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.850287 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.850373 (+ 86us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.850661 (+ 288us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.850853 (+ 192us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:15.850896 (+ 43us) write_op.cc:312] APPLY: finished
0110 20:57:15.851071 (+ 175us) log.cc:844] Serialized 29 byte log entry
0110 20:57:15.851191 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.851247 (+ 56us) write_op.cc:454] Released schema lock
0110 20:57:15.851275 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:15.851469 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.851492 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":327,"apply.run_cpu_time_us":1297,"apply.run_wall_time_us":1297,"num_ops":2,"prepare.queue_time_us":13457,"prepare.run_cpu_time_us":1979,"prepare.run_wall_time_us":171540,"raft.queue_time_us":18575,"raft.run_cpu_time_us":23,"raft.run_wall_time_us":23,"replication_time_us":133329}]]}
W20250110 20:57:15.971923 5399 leader_election.cc:343] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:15.972764 6145 raft_consensus.cc:2798] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:15.973071 6145 raft_consensus.cc:491] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:15.973349 6145 raft_consensus.cc:3054] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:15.950655 4892 rpcz_store.cc:276] 0110 20:57:14.123707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.228277 (+1104570us) service_pool.cc:225] Handling call
0110 20:57:15.229240 (+ 963us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:15.229919 (+ 679us) catalog_manager.cc:3627] Apply alter schema
0110 20:57:15.230573 (+ 654us) catalog_manager.cc:3659] Acquired catalog manager lock
0110 20:57:15.230786 (+ 213us) catalog_manager.cc:3817] Updating metadata on disk
0110 20:57:15.836068 (+605282us) catalog_manager.cc:3848] Committing alterations to in-memory state
0110 20:57:15.950285 (+114217us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:15.374930 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:15.375272 (+ 342us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:15.375293 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:57:15.375306 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:15.375536 (+ 230us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:15.375574 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:15.375595 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:15.375725 (+ 130us) tablet.cc:698] Row locks acquired
0110 20:57:15.375737 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:15.375839 (+ 102us) write_op.cc:270] Start()
0110 20:57:15.375916 (+ 77us) write_op.cc:276] Timestamp: P: 1736542635375772 usec, L: 0
0110 20:57:15.375930 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:15.436698 (+ 60768us) log.cc:844] Serialized 405 byte log entry
0110 20:57:15.514423 (+ 77725us) spinlock_profiling.cc:243] Waited 60.3 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd4796d1 00007f3bfd4f0198 00007f3bfd95c3b1 00007f3bfd95bb88 00007f3bfd960821 00007f3bfd9607d9 00007f3bfd960769 00007f3bfd960731 00007f3bfd95f95d 000000000059ea04 000000000059e839
0110 20:57:15.831647 (+317224us) op_driver.cc:464] REPLICATION: finished
0110 20:57:15.832085 (+ 438us) write_op.cc:301] APPLY: starting
0110 20:57:15.832134 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:15.832223 (+ 89us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:15.832241 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:15.832372 (+ 131us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:15.832493 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:15.832533 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:15.832702 (+ 169us) log.cc:844] Serialized 23 byte log entry
0110 20:57:15.832851 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.832908 (+ 57us) write_op.cc:454] Released schema lock
0110 20:57:15.832935 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:15.833154 (+ 219us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:15.833175 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":300,"apply.run_cpu_time_us":1032,"apply.run_wall_time_us":1032,"num_ops":1,"prepare.queue_time_us":143104,"prepare.run_cpu_time_us":3456,"prepare.run_wall_time_us":139668,"raft.queue_time_us":3748,"raft.run_cpu_time_us":573,"raft.run_wall_time_us":573,"replication_time_us":455627,"spinlock_wait_cycles":138727168}]]}
I20250110 20:57:15.939869 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_bb329a8ba19749ce93ff1d7227aa1fe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:15.975844 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb329a8ba19749ce93ff1d7227aa1fe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:15.978907 6145 raft_consensus.cc:513] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:15.980510 6145 leader_election.cc:290] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:15.943432 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet bae52ea080f54d228b6b76922248ed5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:15.983462 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "491626848ad54be3a18186c65d6dbb60" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:15.983969 5171 raft_consensus.cc:3054] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:15.950135 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f21a12015eab4635890b871de037c960 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:15.876470 5592 tablet_replica.cc:331] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:15.990818 5398 leader_election.cc:343] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 491626848ad54be3a18186c65d6dbb60
I20250110 20:57:15.991295 5171 raft_consensus.cc:2463] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:15.999565 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet fde470279464496aa7efdee3800754f3 (table default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]). Expected version 1 got 0
I20250110 20:57:16.000157 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:16.000957 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:16.006875 6144 raft_consensus.cc:491] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:16.007308 6144 raft_consensus.cc:513] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.007941 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:16.008621 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:16.009596 4891 catalog_manager.cc:5526] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } }
I20250110 20:57:16.010612 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
W20250110 20:57:16.010644 5398 leader_election.cc:343] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 5c5b954dbccb45b2a6ab7cc5cceadffb
I20250110 20:57:16.011121 6144 leader_election.cc:290] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:16.011164 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:16.011624 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:16.015849 5399 leader_election.cc:304] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:16.019194 6145 raft_consensus.cc:2798] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:16.023245 6133 consensus_queue.cc:1035] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:16.033733 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b954dbccb45b2a6ab7cc5cceadffb" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:16.040196 5323 raft_consensus.cc:2463] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:16.047255 5597 tablet_bootstrap.cc:492] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:16.047667 5597 ts_tablet_manager.cc:1397] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.644s user 0.006s sys 0.024s
I20250110 20:57:16.050041 5597 raft_consensus.cc:357] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.050637 5597 raft_consensus.cc:383] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.050904 5597 raft_consensus.cc:738] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.053690 6145 raft_consensus.cc:695] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:16.053972 6200 tablet.cc:1721] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:16.054565 6145 consensus_queue.cc:237] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.055295 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet bae52ea080f54d228b6b76922248ed5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.062355 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet c57941aee12a4ebea2b47f4727fc6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.063282 5398 leader_election.cc:304] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:16.065752 6205 raft_consensus.cc:2798] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:16.066262 6205 raft_consensus.cc:695] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:16.067008 6205 consensus_queue.cc:237] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.070214 5145 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462d21b91c2441fb9e11383466e9ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f8713c1e5a743a489de11bc65857bff) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.070793 6059 raft_consensus.cc:491] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c5bff1e70804ab4a1bb63bdb9345ce6)
I20250110 20:57:16.071228 6059 raft_consensus.cc:513] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:16.074390 5110 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Not found: Tablet not found: 31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:16.082890 5600 tablet_bootstrap.cc:492] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:16.083274 5600 ts_tablet_manager.cc:1397] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.545s user 0.010s sys 0.021s
I20250110 20:57:16.085486 5600 raft_consensus.cc:357] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.086071 5600 raft_consensus.cc:383] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.086340 5600 raft_consensus.cc:738] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.108104 6059 leader_election.cc:290] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:16.108860 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_a9b22870bca84943b105815613f88adb [id=2c78ee52e44a46fe877f474a7ecb771b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet cb7f09df8e274f3e9542292d5c1ec6c6 consensus state has no leader
W20250110 20:57:16.068359 5399 leader_election.cc:343] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.116179 5399 leader_election.cc:304] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:16.116914 5399 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 48580.9us
W20250110 20:57:16.117790 5399 consensus_peers.cc:487] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:16.118872 6129 raft_consensus.cc:2743] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:16.123334 5592 raft_consensus.cc:2238] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:16.124202 5592 raft_consensus.cc:2267] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.126139 5592 ts_tablet_manager.cc:1905] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.113111 5773 ts_tablet_manager.cc:1918] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250110 20:57:16.113637 5111 leader_election.cc:343] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:16.129323 5111 leader_election.cc:304] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:16.131709 6059 raft_consensus.cc:2743] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:16.136679 5773 log.cc:1198] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/e361fb8b47d74548b5a2326d25a9e28c
I20250110 20:57:16.137625 5773 ts_tablet_manager.cc:1939] T e361fb8b47d74548b5a2326d25a9e28c P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:16.139658 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 188) took 1006 ms. Trace:
I20250110 20:57:16.143018 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:16.143344 4893 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_bb329a8ba19749ce93ff1d7227aa1fe6" } modify_external_catalogs: true
I20250110 20:57:16.143795 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:16.143898 4893 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_bb329a8ba19749ce93ff1d7227aa1fe6" } modify_external_catalogs: true
I20250110 20:57:16.144630 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:16.145226 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
W20250110 20:57:16.148150 5399 consensus_peers.cc:487] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:16.148900 4894 master_service.cc:946] ReplaceTablet: received request to replace tablet c44ca47c747544b6afec83b399dcdcb5 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:16.149957 5773 rpcz_store.cc:276] 0110 20:57:15.133731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.133833 (+ 102us) service_pool.cc:225] Handling call
0110 20:57:15.393792 (+259959us) ts_tablet_manager.cc:1145] Deleting tablet e361fb8b47d74548b5a2326d25a9e28c
0110 20:57:15.393828 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:15.869700 (+475872us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.112823 (+243123us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.139581 (+ 26758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":259428}
I20250110 20:57:16.160672 4898 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d40a02bf99094290afe8462ab6a3672f" } new_table_name: "default.Table_0829957c7240427fb7f1fa27fc93cf1c" modify_external_catalogs: true
I20250110 20:57:16.162529 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 184) took 1035 ms. Trace:
I20250110 20:57:16.163069 5773 rpcz_store.cc:276] 0110 20:57:15.127069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.135972 (+ 8903us) service_pool.cc:225] Handling call
0110 20:57:16.162262 (+1026290us) ts_tablet_manager.cc:1145] Deleting tablet 477b09061a4a4f278720168872a83258
0110 20:57:16.162314 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.162483 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025813}
W20250110 20:57:16.147262 5155 consensus_meta.cc:287] T 32cf221df30e4e78bd2a8078c26d29ba P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 0.582s user 0.015s sys 0.004s
I20250110 20:57:16.148775 5597 consensus_queue.cc:260] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.166159 5597 ts_tablet_manager.cc:1428] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.118s user 0.005s sys 0.000s
I20250110 20:57:16.166169 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_36d9c90774f84119abe713d7b6331a49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.167471 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36d9c90774f84119abe713d7b6331a49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.167593 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet e361fb8b47d74548b5a2326d25a9e28c (table default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]) successfully deleted
I20250110 20:57:16.173270 5597 tablet_bootstrap.cc:492] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:16.149439 5144 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 178) took 1244 ms. Trace:
I20250110 20:57:16.177450 5144 rpcz_store.cc:276] 0110 20:57:14.840423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.887293 (+ 46870us) service_pool.cc:225] Handling call
0110 20:57:14.888572 (+ 1279us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:14.888650 (+ 78us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:15.859641 (+970991us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.084702 (+225061us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:16.085127 (+ 425us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:16.180595 5594 tablet_bootstrap.cc:492] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:16.180958 5594 ts_tablet_manager.cc:1397] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.426s user 0.007s sys 0.015s
I20250110 20:57:16.183921 5594 raft_consensus.cc:357] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.184549 5594 raft_consensus.cc:383] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.185823 5594 raft_consensus.cc:738] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.186352 5594 consensus_queue.cc:260] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.198709 6212 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.252:37233 every 8 connection(s)
I20250110 20:57:16.198808 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 190) took 1052 ms. Trace:
I20250110 20:57:16.199069 5773 rpcz_store.cc:276] 0110 20:57:15.146037 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.146132 (+ 95us) service_pool.cc:225] Handling call
0110 20:57:16.198505 (+1052373us) ts_tablet_manager.cc:1145] Deleting tablet d310cf52aa8c4930a7821117a46dc958
0110 20:57:16.198559 (+ 54us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.198748 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051868}
I20250110 20:57:16.171528 4892 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5d22ef7f33a3452db4b509894cf54587" } modify_external_catalogs: true
I20250110 20:57:16.200970 4892 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5d22ef7f33a3452db4b509894cf54587" } modify_external_catalogs: true
I20250110 20:57:16.201201 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 191) took 1053 ms. Trace:
I20250110 20:57:16.201805 5773 rpcz_store.cc:276] 0110 20:57:15.147983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.148078 (+ 95us) service_pool.cc:225] Handling call
0110 20:57:16.200963 (+1052885us) ts_tablet_manager.cc:1145] Deleting tablet d9776df98cb54be38a521852634490e7
0110 20:57:16.201006 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.201149 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052393}
I20250110 20:57:16.202747 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 192) took 1053 ms. Trace:
I20250110 20:57:16.203002 5773 rpcz_store.cc:276] 0110 20:57:15.148876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.148967 (+ 91us) service_pool.cc:225] Handling call
0110 20:57:16.202516 (+1053549us) ts_tablet_manager.cc:1145] Deleting tablet 19d6eca349514b9fb0fff7cd6230027c
0110 20:57:16.202559 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.202703 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053055}
I20250110 20:57:16.203878 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 193) took 1054 ms. Trace:
I20250110 20:57:16.178716 4899 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_57c42d638822484caa513bd1ce8616d9" } new_table_name: "default.Table_eeeff6d7c2e9444bb129b22fe16b8f1c" modify_external_catalogs: true
I20250110 20:57:16.204123 5773 rpcz_store.cc:276] 0110 20:57:15.149739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.149824 (+ 85us) service_pool.cc:225] Handling call
0110 20:57:16.203662 (+1053838us) ts_tablet_manager.cc:1145] Deleting tablet 3a520230d9fd4a74914e46f784dcf747
0110 20:57:16.203702 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.203838 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053351}
I20250110 20:57:16.204895 5773 tablet_replica.cc:331] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:16.205639 5581 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.252:37233
W20250110 20:57:16.208156 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 477b09061a4a4f278720168872a83258 because the tablet was not found. No further retry: Not found: Tablet not found: 477b09061a4a4f278720168872a83258
W20250110 20:57:16.208783 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet d310cf52aa8c4930a7821117a46dc958 because the tablet was not found. No further retry: Not found: Tablet not found: d310cf52aa8c4930a7821117a46dc958
I20250110 20:57:16.209353 5597 tablet_bootstrap.cc:654] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:16.209353 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet d9776df98cb54be38a521852634490e7 because the tablet was not found. No further retry: Not found: Tablet not found: d9776df98cb54be38a521852634490e7
W20250110 20:57:16.210033 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 19d6eca349514b9fb0fff7cd6230027c because the tablet was not found. No further retry: Not found: Tablet not found: 19d6eca349514b9fb0fff7cd6230027c
I20250110 20:57:16.213922 5581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
W20250110 20:57:16.218814 6208 log.cc:926] Time spent T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20250110 20:57:16.222518 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 5581
I20250110 20:57:16.226272 5600 consensus_queue.cc:260] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.228842 5600 ts_tablet_manager.cc:1428] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.145s user 0.005s sys 0.001s
I20250110 20:57:16.236027 6130 raft_consensus.cc:491] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:16.236452 6130 raft_consensus.cc:513] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.238715 6130 leader_election.cc:290] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:16.239009 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637a39e4264b40dd9480dc13e37359d9" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:16.240160 5257 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.241612 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637a39e4264b40dd9480dc13e37359d9" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:16.243222 5255 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.243561 5255 leader_election.cc:304] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:16.244405 6130 raft_consensus.cc:2743] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:16.249722 5773 raft_consensus.cc:2238] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:16.254047 5773 raft_consensus.cc:2267] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:16.256042 4887 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 46018.2us
W20250110 20:57:16.257318 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 3a520230d9fd4a74914e46f784dcf747 because the tablet was not found. No further retry: Not found: Tablet not found: 3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:16.259824 5594 ts_tablet_manager.cc:1428] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.078s user 0.003s sys 0.002s
I20250110 20:57:16.240218 5595 ts_tablet_manager.cc:1918] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:16.260417 5595 log.cc:1198] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/48e53d95ef8645c0bd123138655305ac
I20250110 20:57:16.265344 5595 ts_tablet_manager.cc:1939] T 48e53d95ef8645c0bd123138655305ac P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:16.272533 4895 master_service.cc:946] ReplaceTablet: received request to replace tablet e508061e93cc430cb88f6d10d9684f00 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:16.273922 5773 ts_tablet_manager.cc:1905] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.279595 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 166) took 1811 ms. Trace:
I20250110 20:57:16.280133 5594 tablet_bootstrap.cc:492] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:16.291925 5592 ts_tablet_manager.cc:1918] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:16.305706 5592 log.cc:1198] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/da24904dece04a6194790054f6d7153a
I20250110 20:57:16.306526 5592 ts_tablet_manager.cc:1939] T da24904dece04a6194790054f6d7153a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:16.309031 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.310209 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e43461251413b97721323cd935374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.323346 6195 tablet.cc:1721] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:16.325737 5594 tablet_bootstrap.cc:654] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.327950 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 187) took 1159 ms. Trace:
I20250110 20:57:16.329063 6200 tablet.cc:1721] T a7a09ea54e79485b9892d52d0e04da85 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:16.332211 4893 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 7123604c8fd44a6891e70e14da2d5f28
I20250110 20:57:16.332396 5597 tablet_bootstrap.cc:492] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:16.332767 5597 ts_tablet_manager.cc:1397] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.160s user 0.012s sys 0.002s
I20250110 20:57:16.334944 5597 raft_consensus.cc:357] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:16.335763 4898 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 consensus state has no leader
I20250110 20:57:16.345494 5592 rpcz_store.cc:276] 0110 20:57:15.168305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.170370 (+ 2065us) service_pool.cc:225] Handling call
0110 20:57:15.876286 (+705916us) ts_tablet_manager.cc:1145] Deleting tablet da24904dece04a6194790054f6d7153a
0110 20:57:15.876322 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.201399 (+325077us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.253366 (+ 51967us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.327896 (+ 74530us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":60749,"tablet-delete.queue_time_us":662603}
I20250110 20:57:16.346742 5597 raft_consensus.cc:383] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.346946 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet da24904dece04a6194790054f6d7153a (table default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]) successfully deleted
I20250110 20:57:16.347057 5597 raft_consensus.cc:738] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.347649 5597 consensus_queue.cc:260] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.353706 5592 tablet_replica.cc:331] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:16.354444 5592 raft_consensus.cc:2238] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:16.355443 5597 ts_tablet_manager.cc:1428] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250110 20:57:16.366725 5597 tablet_bootstrap.cc:492] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:16.373804 5595 rpcz_store.cc:276] 0110 20:57:14.468417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.468546 (+ 129us) service_pool.cc:225] Handling call
0110 20:57:15.965633 (+1497087us) ts_tablet_manager.cc:1145] Deleting tablet 48e53d95ef8645c0bd123138655305ac
0110 20:57:15.965674 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.055755 (+ 90081us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.240032 (+184277us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.279543 (+ 39511us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1496519}
I20250110 20:57:16.374819 5595 tablet_replica.cc:331] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:16.374949 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 48e53d95ef8645c0bd123138655305ac (table default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]) successfully deleted
I20250110 20:57:16.375509 5595 raft_consensus.cc:2238] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:16.375953 5595 raft_consensus.cc:2267] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.259560 5600 tablet_bootstrap.cc:492] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:16.333470 6224 tablet.cc:1721] T a7a09ea54e79485b9892d52d0e04da85 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:16.354245 6130 raft_consensus.cc:491] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:16.378726 6130 raft_consensus.cc:513] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.381239 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:16.382632 5257 leader_election.cc:343] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.383759 5597 tablet_bootstrap.cc:654] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.384122 6130 leader_election.cc:290] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:16.370350 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet d310cf52aa8c4930a7821117a46dc958 with delete_type TABLET_DATA_DELETED (replaced by tablet 4aaa42bc5ab042e79baee1ea838851eb) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:16.384291 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:16.384866 5463 raft_consensus.cc:2463] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:16.385900 5255 leader_election.cc:304] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:16.386560 4897 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 41) took 852 ms (client timeout 1000 ms). Trace:
I20250110 20:57:16.386883 6130 raft_consensus.cc:2798] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250110 20:57:16.386880 4897 rpcz_store.cc:269] 0110 20:57:15.534260 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.877799 (+343539us) service_pool.cc:225] Handling call
0110 20:57:16.386497 (+508698us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":18}
I20250110 20:57:16.387254 6130 raft_consensus.cc:491] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250110 20:57:16.388082 4891 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet 371fb7408f0149c8b00659e09ebaf48a consensus state has no leader
I20250110 20:57:16.388262 6130 raft_consensus.cc:3054] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:16.389062 4891 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 59) took 1052 ms (client timeout 1000 ms). Trace:
W20250110 20:57:16.389356 4891 rpcz_store.cc:269] 0110 20:57:15.336347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.337566 (+ 1219us) service_pool.cc:225] Handling call
0110 20:57:16.389004 (+1051438us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:16.050789 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:16.051178 (+ 389us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:16.051201 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:16.051212 (+ 11us) tablet.cc:659] Decoding operations
0110 20:57:16.051456 (+ 244us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:16.051489 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:16.051510 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:16.051644 (+ 134us) tablet.cc:698] Row locks acquired
0110 20:57:16.051657 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:16.051759 (+ 102us) write_op.cc:270] Start()
0110 20:57:16.051837 (+ 78us) write_op.cc:276] Timestamp: P: 1736542636051670 usec, L: 0
0110 20:57:16.051852 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:16.052321 (+ 469us) log.cc:844] Serialized 533 byte log entry
0110 20:57:16.111058 (+ 58737us) op_driver.cc:464] REPLICATION: finished
0110 20:57:16.112823 (+ 1765us) write_op.cc:301] APPLY: starting
0110 20:57:16.112867 (+ 44us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:16.112940 (+ 73us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:16.113029 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:16.113158 (+ 129us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:16.113280 (+ 122us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:16.113319 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:57:16.114168 (+ 849us) log.cc:844] Serialized 23 byte log entry
0110 20:57:16.114295 (+ 127us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:16.114345 (+ 50us) write_op.cc:454] Released schema lock
0110 20:57:16.114371 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:16.114553 (+ 182us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:16.114575 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":130521,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":1669,"apply.run_cpu_time_us":1031,"apply.run_wall_time_us":1694,"num_ops":1,"prepare.queue_time_us":37661,"prepare.run_cpu_time_us":1724,"prepare.run_wall_time_us":1723,"replication_time_us":59145}]]}
I20250110 20:57:16.395375 6130 raft_consensus.cc:513] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.384122 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.394809 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet c951d5ca47e8409db646b43bb04a7034 with delete_type TABLET_DATA_DELETED (replaced by tablet 95b52e115a8e4bae95da0ef9d5afff8c) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.355285 5592 raft_consensus.cc:2267] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.408466 5592 ts_tablet_manager.cc:1905] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.411046 4894 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet c44ca47c747544b6afec83b399dcdcb5
I20250110 20:57:16.412202 4894 catalog_manager.cc:6596] ReplaceTablet: tablet c44ca47c747544b6afec83b399dcdcb5 deleted and replaced by tablet d26c2bd25161489baa156e78128e35aa
I20250110 20:57:16.414637 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet c44ca47c747544b6afec83b399dcdcb5 with delete_type TABLET_DATA_DELETED (replaced by tablet d26c2bd25161489baa156e78128e35aa) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:16.414937 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet c44ca47c747544b6afec83b399dcdcb5 with delete_type TABLET_DATA_DELETED (replaced by tablet d26c2bd25161489baa156e78128e35aa) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:16.415169 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:16.416421 5257 leader_election.cc:343] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.417603 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet c44ca47c747544b6afec83b399dcdcb5 with delete_type TABLET_DATA_DELETED (replaced by tablet d26c2bd25161489baa156e78128e35aa) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.417819 6130 leader_election.cc:290] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:16.400275 5600 tablet_bootstrap.cc:654] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.418016 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:16.418550 5463 raft_consensus.cc:3054] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:16.419559 5597 tablet_bootstrap.cc:492] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:16.419865 5597 ts_tablet_manager.cc:1397] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.001s
I20250110 20:57:16.422077 5597 raft_consensus.cc:357] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.422691 5597 raft_consensus.cc:383] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.422833 6213 sys_catalog.cc:263] Verifying existing consensus state
I20250110 20:57:16.422919 5597 raft_consensus.cc:738] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.400544 5595 ts_tablet_manager.cc:1905] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.425732 5597 consensus_queue.cc:260] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:16.428591 4899 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bb535174548d496e920801723e0909b3 consensus state has no leader
I20250110 20:57:16.429486 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_68433248a92d4f90bf5f613e7722a04c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.430823 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68433248a92d4f90bf5f613e7722a04c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.431442 5597 ts_tablet_manager.cc:1428] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.011s user 0.006s sys 0.001s
I20250110 20:57:16.431478 5463 raft_consensus.cc:2463] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:16.432359 5597 tablet_bootstrap.cc:492] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:16.432646 5255 leader_election.cc:304] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:16.434412 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_43a94d9dc0074febbfe6afa4d4c9716d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.435683 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43a94d9dc0074febbfe6afa4d4c9716d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.448508 5597 tablet_bootstrap.cc:654] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.448324 6130 raft_consensus.cc:2798] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:16.449481 6175 raft_consensus.cc:491] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:16.449867 6175 raft_consensus.cc:513] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.451606 6175 leader_election.cc:290] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:16.452819 5594 tablet_bootstrap.cc:492] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:16.453176 5594 ts_tablet_manager.cc:1397] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.187s user 0.002s sys 0.008s
I20250110 20:57:16.453526 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ef3773a87e4a23a19ac1e5f603d2ce" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:16.455102 5257 leader_election.cc:343] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.456418 6130 raft_consensus.cc:695] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Becoming Leader. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Running, Role: LEADER
I20250110 20:57:16.457262 6130 consensus_queue.cc:237] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.461799 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ef3773a87e4a23a19ac1e5f603d2ce" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:16.463256 5111 consensus_peers.cc:487] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:16.463497 5255 leader_election.cc:343] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.463862 5255 leader_election.cc:304] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:16.464841 6175 raft_consensus.cc:2743] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:16.478659 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_754923f444f045e9b7d6c41c1b11bef0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.480048 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_754923f444f045e9b7d6c41c1b11bef0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.488979 5600 tablet_bootstrap.cc:492] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:16.489490 5600 ts_tablet_manager.cc:1397] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.230s user 0.009s sys 0.002s
I20250110 20:57:16.489820 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet e508061e93cc430cb88f6d10d9684f00
I20250110 20:57:16.491065 4895 catalog_manager.cc:6596] ReplaceTablet: tablet e508061e93cc430cb88f6d10d9684f00 deleted and replaced by tablet e324e1bdd5754864a0f65c50594c2d93
I20250110 20:57:16.491616 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet e508061e93cc430cb88f6d10d9684f00 with delete_type TABLET_DATA_DELETED (replaced by tablet e324e1bdd5754864a0f65c50594c2d93) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:16.492846 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet e508061e93cc430cb88f6d10d9684f00 with delete_type TABLET_DATA_DELETED (replaced by tablet e324e1bdd5754864a0f65c50594c2d93) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:16.472235 5594 raft_consensus.cc:357] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.494828 5594 raft_consensus.cc:383] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.495075 5594 raft_consensus.cc:738] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.498792 6175 consensus_queue.cc:1035] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:16.503594 5773 ts_tablet_manager.cc:1918] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:16.503955 5773 log.cc:1198] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:16.504583 5773 ts_tablet_manager.cc:1939] T 31b65488c884452d876ced1c3d1efb9a P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:16.491639 5600 raft_consensus.cc:357] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.513533 5600 raft_consensus.cc:383] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.513777 5600 raft_consensus.cc:738] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.513854 5594 consensus_queue.cc:260] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.514468 5600 consensus_queue.cc:260] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.516050 5594 ts_tablet_manager.cc:1428] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.063s user 0.004s sys 0.002s
I20250110 20:57:16.516794 5600 ts_tablet_manager.cc:1428] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.027s user 0.004s sys 0.001s
I20250110 20:57:16.527232 5594 tablet_bootstrap.cc:492] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:16.527911 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet e508061e93cc430cb88f6d10d9684f00 with delete_type TABLET_DATA_DELETED (replaced by tablet e324e1bdd5754864a0f65c50594c2d93) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.529943 5600 tablet_bootstrap.cc:492] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:16.535185 5509 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:57:16.535907 5509 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
I20250110 20:57:16.538900 5368 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:57:16.539675 5222 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
W20250110 20:57:16.541020 5257 consensus_peers.cc:487] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:16.546468 6213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Bootstrap starting.
I20250110 20:57:16.548334 5597 tablet_bootstrap.cc:492] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:16.552883 4896 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2fc3f8d84e404d6c85b99d9171a4d645" } new_table_name: "default.Table_5675215f702a41ab934b6baf97fd0431" modify_external_catalogs: true
I20250110 20:57:16.553633 5597 ts_tablet_manager.cc:1397] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.121s user 0.022s sys 0.007s
I20250110 20:57:16.553876 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_99db920d921840a19db98cc103c4fdab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.555191 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99db920d921840a19db98cc103c4fdab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.555684 5597 raft_consensus.cc:357] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.556313 5597 raft_consensus.cc:383] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.556540 5597 raft_consensus.cc:738] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.558501 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 194) took 1408 ms. Trace:
I20250110 20:57:16.558796 5773 rpcz_store.cc:276] 0110 20:57:15.150561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.150646 (+ 85us) service_pool.cc:225] Handling call
0110 20:57:16.204773 (+1054127us) ts_tablet_manager.cc:1145] Deleting tablet 31b65488c884452d876ced1c3d1efb9a
0110 20:57:16.204814 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.339544 (+134730us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.503387 (+163843us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.558446 (+ 55059us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29719,"tablet-delete.queue_time_us":1053651}
I20250110 20:57:16.559934 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 31b65488c884452d876ced1c3d1efb9a (table default.Table_d29551acec544d57b14d18fecbf2722e [id=16478fe1cba54c88ba8245c365d5a992]) successfully deleted
I20250110 20:57:16.562294 4895 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d08622be3a1c4350b1df15c8f949733d" } modify_external_catalogs: true
I20250110 20:57:16.562752 4895 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_d08622be3a1c4350b1df15c8f949733d" } modify_external_catalogs: true
I20250110 20:57:16.566138 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 195) took 1414 ms. Trace:
I20250110 20:57:16.566429 5773 rpcz_store.cc:276] 0110 20:57:15.151373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.151457 (+ 84us) service_pool.cc:225] Handling call
0110 20:57:16.560590 (+1409133us) ts_tablet_manager.cc:1145] Deleting tablet ae5956b1aa17449497f9695a111ebb66
0110 20:57:16.565871 (+ 5281us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.566085 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408683}
W20250110 20:57:16.567549 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet ae5956b1aa17449497f9695a111ebb66 because the tablet was not found. No further retry: Not found: Tablet not found: ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:16.557066 5597 consensus_queue.cc:260] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.575680 5597 ts_tablet_manager.cc:1428] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20250110 20:57:16.580817 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 196) took 1428 ms. Trace:
I20250110 20:57:16.581113 5773 rpcz_store.cc:276] 0110 20:57:15.152189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.152278 (+ 89us) service_pool.cc:225] Handling call
0110 20:57:16.568229 (+1415951us) ts_tablet_manager.cc:1145] Deleting tablet f1fafed70adb496d97acfd5fea356292
0110 20:57:16.575940 (+ 7711us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.580766 (+ 4826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1415491}
W20250110 20:57:16.582186 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet f1fafed70adb496d97acfd5fea356292 because the tablet was not found. No further retry: Not found: Tablet not found: f1fafed70adb496d97acfd5fea356292
I20250110 20:57:16.583149 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 189) took 1437 ms. Trace:
I20250110 20:57:16.583442 5773 rpcz_store.cc:276] 0110 20:57:15.145758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.149862 (+ 4104us) service_pool.cc:225] Handling call
0110 20:57:16.582866 (+1433004us) ts_tablet_manager.cc:1145] Deleting tablet ae5956b1aa17449497f9695a111ebb66
0110 20:57:16.582918 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.583099 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1429597}
W20250110 20:57:16.584424 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet ae5956b1aa17449497f9695a111ebb66 because the tablet was not found. No further retry: Not found: Tablet not found: ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:16.590545 5595 ts_tablet_manager.cc:1918] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:16.590888 5595 log.cc:1198] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/d1afe41a09954fc8873ea53a330ea03f
I20250110 20:57:16.591519 5595 ts_tablet_manager.cc:1939] T d1afe41a09954fc8873ea53a330ea03f P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:16.594190 5597 tablet_bootstrap.cc:492] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:16.603099 6173 consensus_queue.cc:1035] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:16.609197 5594 tablet_bootstrap.cc:654] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.593320 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 167) took 2040 ms. Trace:
I20250110 20:57:16.621737 5595 rpcz_store.cc:276] 0110 20:57:14.553306 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:14.553747 (+ 441us) service_pool.cc:225] Handling call
0110 20:57:16.374683 (+1820936us) ts_tablet_manager.cc:1145] Deleting tablet d1afe41a09954fc8873ea53a330ea03f
0110 20:57:16.374723 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.438350 (+ 63627us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.590317 (+151967us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.593261 (+ 2944us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1820400}
I20250110 20:57:16.622961 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet d1afe41a09954fc8873ea53a330ea03f (table default.Table_2aad5f88cd6e4aaf953eb9cc06e46c6c [id=0f2abb2b06c2485aab55a063ad4a0b47]) successfully deleted
I20250110 20:57:16.623831 5595 tablet_replica.cc:331] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:16.624567 5595 raft_consensus.cc:2238] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:16.632154 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_31bd404aec6f4df489bae22a80054bfc" } new_table_name: "default.Table_b2bea6d7461644019524e9f4455cde2e" modify_external_catalogs: true
I20250110 20:57:16.636168 5592 ts_tablet_manager.cc:1918] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:16.636543 5592 log.cc:1198] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/2967c06715e241dcbbe784eed2373704
I20250110 20:57:16.637164 5592 ts_tablet_manager.cc:1939] T 2967c06715e241dcbbe784eed2373704 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:16.638965 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 189) took 1470 ms. Trace:
I20250110 20:57:16.639245 5592 rpcz_store.cc:276] 0110 20:57:15.168897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.171017 (+ 2120us) service_pool.cc:225] Handling call
0110 20:57:16.353572 (+1182555us) ts_tablet_manager.cc:1145] Deleting tablet 2967c06715e241dcbbe784eed2373704
0110 20:57:16.353610 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.527670 (+174060us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.635958 (+108288us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.638917 (+ 2959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1139468}
I20250110 20:57:16.640280 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 2967c06715e241dcbbe784eed2373704 (table default.Table_6d29a7293ace46b186d68a71d0a5a688 [id=4f67b22c80ae49b79ddc245c7e304307]) successfully deleted
I20250110 20:57:16.616683 5773 tablet_replica.cc:331] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
W20250110 20:57:16.646245 4896 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7309d13f62524b6ab8b4c551728ecb6d consensus state has no leader
I20250110 20:57:16.653978 5773 raft_consensus.cc:2238] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:16.654542 5773 raft_consensus.cc:2267] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.655161 5592 tablet_replica.cc:331] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:16.656385 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76bedccd828042948a702fe840788765 consensus state has no leader
I20250110 20:57:16.659107 5597 tablet_bootstrap.cc:654] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.655849 5592 raft_consensus.cc:2238] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:16.670209 5592 raft_consensus.cc:2267] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.656435 5773 ts_tablet_manager.cc:1905] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.674568 5595 raft_consensus.cc:2267] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.681883 5592 ts_tablet_manager.cc:1905] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.678750 6232 consensus_queue.cc:1035] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20250110 20:57:16.686108 5595 ts_tablet_manager.cc:1905] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.695958 5600 tablet_bootstrap.cc:654] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.697578 6205 raft_consensus.cc:491] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:16.697963 6205 raft_consensus.cc:513] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.705819 6205 leader_election.cc:290] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:16.707129 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc7b1acf4114bf69d801437855760df" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:16.708092 5398 leader_election.cc:343] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: fcc7b1acf4114bf69d801437855760df
W20250110 20:57:16.708618 5399 leader_election.cc:343] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.708943 5399 leader_election.cc:304] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:16.709901 6205 raft_consensus.cc:2743] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:16.719946 5595 ts_tablet_manager.cc:1918] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:16.720295 5595 log.cc:1198] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/e27e43461251413b97721323cd935374
I20250110 20:57:16.720937 5595 ts_tablet_manager.cc:1939] T e27e43461251413b97721323cd935374 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:16.722971 5597 tablet_bootstrap.cc:492] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:16.723359 5597 ts_tablet_manager.cc:1397] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.129s user 0.008s sys 0.000s
I20250110 20:57:16.732309 6232 consensus_queue.cc:1035] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:16.737208 5597 raft_consensus.cc:357] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.737862 5597 raft_consensus.cc:383] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.738121 5597 raft_consensus.cc:738] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.743026 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 177) took 1705 ms. Trace:
I20250110 20:57:16.743460 4894 master_service.cc:946] ReplaceTablet: received request to replace tablet e508061e93cc430cb88f6d10d9684f00 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:16.743347 5595 rpcz_store.cc:276] 0110 20:57:15.037976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.039151 (+ 1175us) service_pool.cc:225] Handling call
0110 20:57:16.623697 (+1584546us) ts_tablet_manager.cc:1145] Deleting tablet e27e43461251413b97721323cd935374
0110 20:57:16.623735 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.700393 (+ 76658us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.719760 (+ 19367us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.742978 (+ 23218us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49410,"tablet-delete.queue_time_us":1584010}
I20250110 20:57:16.744614 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet e27e43461251413b97721323cd935374 (table default.Table_ead31c28470a4d3384b767df0ecce1ef [id=729d5b2683d04c36ac582ea2965304de]) successfully deleted
I20250110 20:57:16.744956 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 178) took 1627 ms. Trace:
I20250110 20:57:16.743886 5597 consensus_queue.cc:260] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.758720 5597 ts_tablet_manager.cc:1428] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.035s user 0.004s sys 0.002s
I20250110 20:57:16.759577 5597 tablet_bootstrap.cc:492] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:16.745218 5595 rpcz_store.cc:276] 0110 20:57:15.117315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.117708 (+ 393us) service_pool.cc:225] Handling call
0110 20:57:16.744734 (+1627026us) ts_tablet_manager.cc:1145] Deleting tablet 491626848ad54be3a18186c65d6dbb60
0110 20:57:16.744770 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.744909 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1617911}
W20250110 20:57:16.761355 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 491626848ad54be3a18186c65d6dbb60 because the tablet was not found. No further retry: Not found: Tablet not found: 491626848ad54be3a18186c65d6dbb60
I20250110 20:57:16.764086 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a262069587954407a948374ed4c1b18f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:16.764817 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_8e3dc8862ef2488b9fe7b37dfc1f3a09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.765599 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a262069587954407a948374ed4c1b18f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:16.766113 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e3dc8862ef2488b9fe7b37dfc1f3a09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.768862 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e7d42f471e7a4c3999d9b4ed548ae2f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.770174 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d42f471e7a4c3999d9b4ed548ae2f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.790939 5773 ts_tablet_manager.cc:1918] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:16.794416 5600 tablet_bootstrap.cc:492] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:16.794832 5600 ts_tablet_manager.cc:1397] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.265s user 0.007s sys 0.004s
I20250110 20:57:16.798732 6213 log.cc:826] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Log is configured to *not* fsync() on all Append() calls
I20250110 20:57:16.805555 5773 log.cc:1198] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/2967c06715e241dcbbe784eed2373704
I20250110 20:57:16.806372 5773 ts_tablet_manager.cc:1939] T 2967c06715e241dcbbe784eed2373704 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:16.808600 5597 tablet_bootstrap.cc:654] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:16.809070 4900 catalog_manager.cc:5526] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:16.812709 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:16.814667 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:16.823999 5509 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
I20250110 20:57:16.826676 5594 tablet_bootstrap.cc:492] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:16.827069 5594 ts_tablet_manager.cc:1397] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.300s user 0.008s sys 0.010s
I20250110 20:57:16.773986 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 180) took 1656 ms. Trace:
I20250110 20:57:16.818192 4894 master_service.cc:946] ReplaceTablet: received request to replace tablet c951d5ca47e8409db646b43bb04a7034 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:16.830286 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:16.830297 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 62fe5d3bd809412a8bce721a57406f84 (DEFAULT_TABLE table=default.Table_634a503e9332481eb24fe3aa1a7c37a2 [id=82b804310c184b63bbc27b249f912283]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:16.831805 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62fe5d3bd809412a8bce721a57406f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:16.835062 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 199) took 1658 ms. Trace:
I20250110 20:57:16.835398 5773 rpcz_store.cc:276] 0110 20:57:15.177129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.177225 (+ 96us) service_pool.cc:225] Handling call
0110 20:57:16.616541 (+1439316us) ts_tablet_manager.cc:1145] Deleting tablet 2967c06715e241dcbbe784eed2373704
0110 20:57:16.616589 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.739520 (+122931us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.790738 (+ 51218us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.835010 (+ 44272us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1438735}
I20250110 20:57:16.837814 5773 tablet_replica.cc:331] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:16.838493 5773 raft_consensus.cc:2238] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:16.839309 5773 pending_rounds.cc:62] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:16.845484 5773 pending_rounds.cc:69] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878619599720448 op_type: NO_OP noop_request { }
I20250110 20:57:16.845880 5773 raft_consensus.cc:2883] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:16.846201 5773 raft_consensus.cc:2267] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.847934 5773 ts_tablet_manager.cc:1905] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:16.831101 5595 rpcz_store.cc:276] 0110 20:57:15.117838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.153220 (+ 35382us) service_pool.cc:225] Handling call
0110 20:57:16.773720 (+1620500us) ts_tablet_manager.cc:1145] Deleting tablet ae5956b1aa17449497f9695a111ebb66
0110 20:57:16.773763 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.773933 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606900}
I20250110 20:57:16.854287 5595 tablet_replica.cc:331] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:16.855068 5595 raft_consensus.cc:2238] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:16.855666 5595 raft_consensus.cc:2267] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:16.837152 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 2967c06715e241dcbbe784eed2373704 (table default.Table_6d29a7293ace46b186d68a71d0a5a688 [id=4f67b22c80ae49b79ddc245c7e304307]) successfully deleted
I20250110 20:57:16.854007 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_c7eb26eb1ef541e7853e3445290e6d49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:16.858944 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7eb26eb1ef541e7853e3445290e6d49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:16.859045 4891 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a9b22870bca84943b105815613f88adb" } new_table_name: "default.Table_1588a911bebe4556aee77ec7ebb160cf" modify_external_catalogs: true
W20250110 20:57:16.855104 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet ae5956b1aa17449497f9695a111ebb66 because the tablet was not found. No further retry: Not found: Tablet not found: ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:16.856046 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_3c32fcd3cede42049db2626e1952ecb4" } new_table_name: "default.Table_667604fe371a4f43a1c96e082923f62a" modify_external_catalogs: true
I20250110 20:57:16.880815 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet a95f05fc5afe4877a606b501dbd09551
I20250110 20:57:16.884274 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet a95f05fc5afe4877a606b501dbd09551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:16.885768 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet a95f05fc5afe4877a606b501dbd09551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:16.886471 5595 ts_tablet_manager.cc:1905] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:16.909130 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79e065ae748a41d68ad1abe0471b5ed9 consensus state has no leader
I20250110 20:57:16.899879 5592 ts_tablet_manager.cc:1918] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:16.918149 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:16.918963 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
I20250110 20:57:16.919663 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_a9b22870bca84943b105815613f88adb [id=2c78ee52e44a46fe877f474a7ecb771b]). Expected version 1 got 0
I20250110 20:57:16.918524 5592 log.cc:1198] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:16.921965 5592 ts_tablet_manager.cc:1939] T 8d67e94c8f9a414499b5e20a5cc0ead3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:16.923735 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 62fe5d3bd809412a8bce721a57406f84 (DEFAULT_TABLE table=default.Table_634a503e9332481eb24fe3aa1a7c37a2 [id=82b804310c184b63bbc27b249f912283]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:16.925567 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62fe5d3bd809412a8bce721a57406f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:16.925534 5155 tablet_service.cc:1467] Processing CreateTablet for tablet a95f05fc5afe4877a606b501dbd09551 (DEFAULT_TABLE table=default.Table_d08622be3a1c4350b1df15c8f949733d [id=f3959fdb299c4e5698a5cb602e8d1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:16.926545 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95f05fc5afe4877a606b501dbd09551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:16.927331 6175 raft_consensus.cc:491] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:16.927745 6175 raft_consensus.cc:513] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.929538 6175 leader_election.cc:290] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:16.930881 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e508061e93cc430cb88f6d10d9684f00" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:16.931468 5600 raft_consensus.cc:357] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.932101 5600 raft_consensus.cc:383] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.932341 5600 raft_consensus.cc:738] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:16.932547 5257 leader_election.cc:343] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:16.939175 5600 consensus_queue.cc:260] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.939929 5294 tablet_service.cc:1467] Processing CreateTablet for tablet a95f05fc5afe4877a606b501dbd09551 (DEFAULT_TABLE table=default.Table_d08622be3a1c4350b1df15c8f949733d [id=f3959fdb299c4e5698a5cb602e8d1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:16.940943 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95f05fc5afe4877a606b501dbd09551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:16.944826 5600 ts_tablet_manager.cc:1428] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.150s user 0.006s sys 0.000s
I20250110 20:57:16.946512 6172 raft_consensus.cc:491] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:16.951565 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 16aaabfff9a04310b19a3195d0b41d7a (DEFAULT_TABLE table=default.Table_461c0135de1f4d318a4f6ceaf204ae45 [id=aa4e3169f5294e5e8ee337f07b08e422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:16.952504 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16aaabfff9a04310b19a3195d0b41d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:16.967495 5597 tablet_bootstrap.cc:492] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:16.967864 5597 ts_tablet_manager.cc:1397] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.208s user 0.004s sys 0.010s
I20250110 20:57:16.970026 5597 raft_consensus.cc:357] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.970624 5597 raft_consensus.cc:383] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:16.970867 5597 raft_consensus.cc:738] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:16.946921 6172 raft_consensus.cc:513] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.976241 6172 leader_election.cc:290] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:16.977634 5597 consensus_queue.cc:260] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:16.978559 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 190) took 1809 ms. Trace:
I20250110 20:57:16.978896 5592 rpcz_store.cc:276] 0110 20:57:15.169221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.171216 (+ 1995us) service_pool.cc:225] Handling call
0110 20:57:16.655035 (+1483819us) ts_tablet_manager.cc:1145] Deleting tablet 8d67e94c8f9a414499b5e20a5cc0ead3
0110 20:57:16.655070 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.858426 (+203356us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.899530 (+ 41104us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:16.978504 (+ 78974us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1431645}
I20250110 20:57:16.979698 5597 ts_tablet_manager.cc:1428] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.012s user 0.004s sys 0.002s
I20250110 20:57:16.980289 5592 tablet_replica.cc:331] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:16.980468 5597 tablet_bootstrap.cc:492] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:16.980751 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 16aaabfff9a04310b19a3195d0b41d7a (DEFAULT_TABLE table=default.Table_461c0135de1f4d318a4f6ceaf204ae45 [id=aa4e3169f5294e5e8ee337f07b08e422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:16.981612 5592 raft_consensus.cc:2238] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:16.982187 6175 raft_consensus.cc:491] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 20:57:16.982823 5111 leader_election.cc:343] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: cd0a666892d04e97a4b9967e38ab0946
I20250110 20:57:16.982607 6175 raft_consensus.cc:513] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:16.984354 6175 leader_election.cc:290] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:16.977233 6213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=6 ignored=0} inserts{seen=7 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250110 20:57:16.987133 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:16.987391 6213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: Bootstrap complete.
I20250110 20:57:16.988332 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16aaabfff9a04310b19a3195d0b41d7a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:16.988998 5257 leader_election.cc:343] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:17.002348 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_a9b22870bca84943b105815613f88adb [id=2c78ee52e44a46fe877f474a7ecb771b]). Expected version 1 got 0
I20250110 20:57:17.003228 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
I20250110 20:57:17.010751 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (table default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145]). Expected version 1 got 0
I20250110 20:57:17.011425 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:17.012069 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet a7a09ea54e79485b9892d52d0e04da85 (table default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76]). Expected version 2 got 0
I20250110 20:57:17.012702 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:17.013466 4898 catalog_manager.cc:5526] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194). New cstate: current_term: 1 leader_uuid: "46005a5920cc4a3697a5c61b1c271384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:17.015192 5597 tablet_bootstrap.cc:654] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:17.015393 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_a9b22870bca84943b105815613f88adb [id=2c78ee52e44a46fe877f474a7ecb771b]). Expected version 1 got 0
I20250110 20:57:17.015986 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:17.017091 4898 catalog_manager.cc:5526] T bd9d0249798e4f24adf0b36d9e059b7e P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194). New cstate: current_term: 1 leader_uuid: "46005a5920cc4a3697a5c61b1c271384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:17.020998 5144 tablet_service.cc:1467] Processing CreateTablet for tablet c7961f8bd3ff42ffab214a04c704fd26 (DEFAULT_TABLE table=default.Table_1709c1058a134005a1475580de97779b [id=cf5d35e4dfbf45688b9ba58240ac8262]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.022025 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7961f8bd3ff42ffab214a04c704fd26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.033397 5592 raft_consensus.cc:2267] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.036209 5592 ts_tablet_manager.cc:1905] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.048141 5600 tablet_bootstrap.cc:492] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:17.051957 5293 tablet_service.cc:1467] Processing CreateTablet for tablet c7961f8bd3ff42ffab214a04c704fd26 (DEFAULT_TABLE table=default.Table_1709c1058a134005a1475580de97779b [id=cf5d35e4dfbf45688b9ba58240ac8262]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.058257 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 79e065ae748a41d68ad1abe0471b5ed9 (DEFAULT_TABLE table=default.Table_667604fe371a4f43a1c96e082923f62a [id=19e4e1b23fd74c278cad78decb74aaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.058876 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:17.059864 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e065ae748a41d68ad1abe0471b5ed9. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:17.060068 5255 leader_election.cc:343] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:17.060487 5255 leader_election.cc:304] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:17.061133 6175 raft_consensus.cc:2743] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:17.047698 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e508061e93cc430cb88f6d10d9684f00" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:17.070874 5255 leader_election.cc:343] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:17.071416 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a666892d04e97a4b9967e38ab0946" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:17.071415 5255 leader_election.cc:304] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:17.071391 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 79e065ae748a41d68ad1abe0471b5ed9 (DEFAULT_TABLE table=default.Table_667604fe371a4f43a1c96e082923f62a [id=19e4e1b23fd74c278cad78decb74aaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.071964 5463 raft_consensus.cc:2463] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:17.072108 6175 raft_consensus.cc:2743] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:17.072907 5773 ts_tablet_manager.cc:1918] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:17.073303 5773 log.cc:1198] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:17.073663 5110 leader_election.cc:304] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:17.074045 5773 ts_tablet_manager.cc:1939] T 1ac07ac6fc9f477ba7f9accc5533be39 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:17.043012 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e3046e912cd243e591470e603e86c4d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:17.075129 6172 raft_consensus.cc:2798] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250110 20:57:17.075204 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3046e912cd243e591470e603e86c4d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.075450 6172 raft_consensus.cc:491] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:17.075736 6172 raft_consensus.cc:3054] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.046077 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet 01962c5b7ae64f80b998948d18250b4c from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:17.079010 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 200) took 1901 ms. Trace:
I20250110 20:57:17.079347 5773 rpcz_store.cc:276] 0110 20:57:15.177425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.177510 (+ 85us) service_pool.cc:225] Handling call
0110 20:57:16.837666 (+1660156us) ts_tablet_manager.cc:1145] Deleting tablet 1ac07ac6fc9f477ba7f9accc5533be39
0110 20:57:16.837721 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.001903 (+164182us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.072710 (+ 70807us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.078957 (+ 6247us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3114,"tablet-delete.queue_time_us":1659673}
I20250110 20:57:17.080478 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 197) took 1904 ms. Trace:
I20250110 20:57:17.047287 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_bdd0ff0e681b4f57ba64a67252186056"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:17.080806 5773 rpcz_store.cc:276] 0110 20:57:15.176518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.177543 (+ 1025us) service_pool.cc:225] Handling call
0110 20:57:17.080234 (+1902691us) ts_tablet_manager.cc:1145] Deleting tablet 1ac07ac6fc9f477ba7f9accc5533be39
0110 20:57:17.080279 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.080437 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1901456}
I20250110 20:57:17.081631 5773 tablet_replica.cc:331] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
W20250110 20:57:17.081681 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdd0ff0e681b4f57ba64a67252186056 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.082218 5773 raft_consensus.cc:2238] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:17.082711 5773 raft_consensus.cc:2267] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.050335 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_4a1acf754c984e86ac40fd7ab1509b2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:17.085512 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a1acf754c984e86ac40fd7ab1509b2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.086617 5773 ts_tablet_manager.cc:1905] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.087415 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7961f8bd3ff42ffab214a04c704fd26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.092018 6059 raft_consensus.cc:491] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.092512 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 1ac07ac6fc9f477ba7f9accc5533be39 (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]) successfully deleted
I20250110 20:57:17.092425 6059 raft_consensus.cc:513] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:17.093119 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 1ac07ac6fc9f477ba7f9accc5533be39 because the tablet was not found. No further retry: Not found: Tablet not found: 1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:17.094167 6059 leader_election.cc:290] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:17.096385 5594 raft_consensus.cc:357] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.097033 5594 raft_consensus.cc:383] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:17.100379 5111 leader_election.cc:343] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 2c1ca1670916401ab0a43685df1b6759
I20250110 20:57:17.102953 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1ca1670916401ab0a43685df1b6759" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:17.103520 5463 raft_consensus.cc:2463] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:17.104595 5110 leader_election.cc:304] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:17.105511 6059 raft_consensus.cc:2798] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:17.105831 6059 raft_consensus.cc:491] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:17.106115 6059 raft_consensus.cc:3054] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.097273 5594 raft_consensus.cc:738] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:17.108309 5594 consensus_queue.cc:260] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.105183 6172 raft_consensus.cc:513] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.114881 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 01962c5b7ae64f80b998948d18250b4c
I20250110 20:57:17.114894 6172 leader_election.cc:290] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:17.115793 5111 leader_election.cc:343] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: cd0a666892d04e97a4b9967e38ab0946
I20250110 20:57:17.116298 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd0a666892d04e97a4b9967e38ab0946" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:17.116539 4896 catalog_manager.cc:6596] ReplaceTablet: tablet 01962c5b7ae64f80b998948d18250b4c deleted and replaced by tablet b5becf70b58847559b70ce27edc30398
I20250110 20:57:17.116823 5463 raft_consensus.cc:3054] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.119149 5148 tablet_service.cc:1514] Processing DeleteTablet for tablet 01962c5b7ae64f80b998948d18250b4c with delete_type TABLET_DATA_DELETED (replaced by tablet b5becf70b58847559b70ce27edc30398) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:17.066506 5595 ts_tablet_manager.cc:1918] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:17.126261 5595 log.cc:1198] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/637a39e4264b40dd9480dc13e37359d9
I20250110 20:57:17.127444 5595 ts_tablet_manager.cc:1939] T 637a39e4264b40dd9480dc13e37359d9 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:17.117374 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1709c1058a134005a1475580de97779b" } modify_external_catalogs: true
W20250110 20:57:17.125751 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_5675215f702a41ab934b6baf97fd0431 [id=1c2b9e8ff5d341c380b50023c577d0aa], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7309d13f62524b6ab8b4c551728ecb6d consensus state has no leader
I20250110 20:57:17.120005 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_634a503e9332481eb24fe3aa1a7c37a2" } new_table_name: "default.Table_76e5da9e2e7e4e58a06bc9d08ffbd8c2" modify_external_catalogs: true
I20250110 20:57:17.130568 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 181) took 2012 ms. Trace:
I20250110 20:57:17.130877 6059 raft_consensus.cc:513] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.134160 5594 ts_tablet_manager.cc:1428] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.307s user 0.006s sys 0.001s
I20250110 20:57:17.137533 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1709c1058a134005a1475580de97779b" } modify_external_catalogs: true
I20250110 20:57:17.150480 5594 tablet_bootstrap.cc:492] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:17.146330 6172 raft_consensus.cc:491] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.151443 6172 raft_consensus.cc:513] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.154440 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c1ca1670916401ab0a43685df1b6759" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:17.154978 5460 raft_consensus.cc:3054] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.155357 5463 raft_consensus.cc:2463] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:17.156383 5110 leader_election.cc:304] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
W20250110 20:57:17.158089 5111 leader_election.cc:343] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 2c1ca1670916401ab0a43685df1b6759
I20250110 20:57:17.163030 5460 raft_consensus.cc:2463] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:17.163801 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_504da40a6cc54bb4b377ba25e145e4e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:17.165349 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_504da40a6cc54bb4b377ba25e145e4e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:17.166137 5111 leader_election.cc:343] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: c6914e258314423c859eb31e85853274
I20250110 20:57:17.168042 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 8d67e94c8f9a414499b5e20a5cc0ead3 (table default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]) successfully deleted
I20250110 20:57:17.173120 5773 ts_tablet_manager.cc:1918] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:17.173612 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 01962c5b7ae64f80b998948d18250b4c with delete_type TABLET_DATA_DELETED (replaced by tablet b5becf70b58847559b70ce27edc30398) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:17.131156 5595 rpcz_store.cc:276] 0110 20:57:15.118100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.153492 (+ 35392us) service_pool.cc:225] Handling call
0110 20:57:16.854138 (+1700646us) ts_tablet_manager.cc:1145] Deleting tablet 637a39e4264b40dd9480dc13e37359d9
0110 20:57:16.854179 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:16.949098 (+ 94919us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.066128 (+117030us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.130485 (+ 64357us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7087,"tablet-delete.queue_time_us":1685386}
I20250110 20:57:17.178517 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 179) took 2060 ms. Trace:
I20250110 20:57:17.178802 5595 rpcz_store.cc:276] 0110 20:57:15.117591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.160559 (+ 42968us) service_pool.cc:225] Handling call
0110 20:57:17.178275 (+2017716us) ts_tablet_manager.cc:1145] Deleting tablet 5c5b954dbccb45b2a6ab7cc5cceadffb
0110 20:57:17.178314 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.178464 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5638,"tablet-delete.queue_time_us":2007974}
I20250110 20:57:17.179774 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 187) took 1538 ms. Trace:
I20250110 20:57:17.180043 5595 rpcz_store.cc:276] 0110 20:57:15.641112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.641245 (+ 133us) service_pool.cc:225] Handling call
0110 20:57:17.179570 (+1538325us) ts_tablet_manager.cc:1145] Deleting tablet 19d6eca349514b9fb0fff7cd6230027c
0110 20:57:17.179605 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.179730 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1537791}
I20250110 20:57:17.181972 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 189) took 1406 ms. Trace:
I20250110 20:57:17.183241 5592 ts_tablet_manager.cc:1918] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:17.183142 5595 rpcz_store.cc:276] 0110 20:57:15.775974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.776098 (+ 124us) service_pool.cc:225] Handling call
0110 20:57:17.181779 (+1405681us) ts_tablet_manager.cc:1145] Deleting tablet fcc7b1acf4114bf69d801437855760df
0110 20:57:17.181810 (+ 31us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.181928 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1405132}
I20250110 20:57:17.184020 5595 tablet_replica.cc:331] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:17.184348 5592 log.cc:1198] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:17.185652 5592 ts_tablet_manager.cc:1939] T 7cf0c87159574bfcbc7a34abe508f9d6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:17.184710 5595 raft_consensus.cc:2238] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:17.191323 5595 raft_consensus.cc:2267] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.192885 5595 ts_tablet_manager.cc:1905] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.194160 5773 log.cc:1198] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/da24904dece04a6194790054f6d7153a
I20250110 20:57:17.194839 5773 ts_tablet_manager.cc:1939] T da24904dece04a6194790054f6d7153a P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:17.195623 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 637a39e4264b40dd9480dc13e37359d9 (table default.Table_638a8034373e4e3bbfb842beaacdaab8 [id=42aa140491c94323b9a140eea45f28de]) successfully deleted
W20250110 20:57:17.196436 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb because the tablet was not found. No further retry: Not found: Tablet not found: 5c5b954dbccb45b2a6ab7cc5cceadffb
W20250110 20:57:17.197283 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 19d6eca349514b9fb0fff7cd6230027c because the tablet was not found. No further retry: Not found: Tablet not found: 19d6eca349514b9fb0fff7cd6230027c
W20250110 20:57:17.198128 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet fcc7b1acf4114bf69d801437855760df because the tablet was not found. No further retry: Not found: Tablet not found: fcc7b1acf4114bf69d801437855760df
I20250110 20:57:17.201998 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
I20250110 20:57:17.202700 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:17.180760 6059 leader_election.cc:290] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:17.203897 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 47594.7us
I20250110 20:57:17.204578 5110 leader_election.cc:304] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:17.208837 5303 tablet_service.cc:1467] Processing CreateTablet for tablet ba0d83a50ec0479f9673a57370b0e879 (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.191363 6205 raft_consensus.cc:491] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.209834 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0d83a50ec0479f9673a57370b0e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.209784 6205 raft_consensus.cc:513] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.210902 5144 tablet_service.cc:1467] Processing CreateTablet for tablet ba0d83a50ec0479f9673a57370b0e879 (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.211524 6205 leader_election.cc:290] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:17.211789 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0d83a50ec0479f9673a57370b0e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.195137 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet a95f05fc5afe4877a606b501dbd09551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:17.196305 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 62fe5d3bd809412a8bce721a57406f84 (DEFAULT_TABLE table=default.Table_634a503e9332481eb24fe3aa1a7c37a2 [id=82b804310c184b63bbc27b249f912283]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.197644 5438 tablet_service.cc:1467] Processing CreateTablet for tablet a95f05fc5afe4877a606b501dbd09551 (DEFAULT_TABLE table=default.Table_d08622be3a1c4350b1df15c8f949733d [id=f3959fdb299c4e5698a5cb602e8d1e0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.213371 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62fe5d3bd809412a8bce721a57406f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.214366 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a95f05fc5afe4877a606b501dbd09551. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:17.214190 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62fe5d3bd809412a8bce721a57406f84 consensus state has no leader
I20250110 20:57:17.224057 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:17.224581 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff]). Expected version 1 got 0
I20250110 20:57:17.226743 5597 tablet_bootstrap.cc:492] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:17.227159 5597 ts_tablet_manager.cc:1397] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.247s user 0.009s sys 0.000s
I20250110 20:57:17.227470 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba1b57de9e4b4b2ab6c3bce8fe78166a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:17.228359 5176 raft_consensus.cc:2463] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:17.228729 6172 leader_election.cc:290] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:17.229204 6172 raft_consensus.cc:2798] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:17.229604 6172 raft_consensus.cc:695] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:17.230314 6172 consensus_queue.cc:237] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.237829 6253 raft_consensus.cc:2798] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:17.240579 6205 raft_consensus.cc:491] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.241086 6205 raft_consensus.cc:513] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.242503 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 198) took 2065 ms. Trace:
I20250110 20:57:17.243022 6205 leader_election.cc:290] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:17.244272 5773 rpcz_store.cc:276] 0110 20:57:15.176839 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.177691 (+ 852us) service_pool.cc:225] Handling call
0110 20:57:17.081502 (+1903811us) ts_tablet_manager.cc:1145] Deleting tablet da24904dece04a6194790054f6d7153a
0110 20:57:17.081544 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.143608 (+ 62064us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.172924 (+ 29316us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.242444 (+ 69520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1902546}
I20250110 20:57:17.245430 5773 tablet_replica.cc:331] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:17.245611 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet da24904dece04a6194790054f6d7153a (table default.Table_26dcb5fd161748d3897adbcc67f697e2 [id=ecb8d647dcaa491baf92fca4e1399597]) successfully deleted
I20250110 20:57:17.255705 5600 tablet_bootstrap.cc:654] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:17.255667 5773 raft_consensus.cc:2238] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:17.256733 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6229e4efd84e43a77cf359abdaa648 (DEFAULT_TABLE table=default.Table_e73d9dd6e1d44352936cc594ddd9de99 [id=61645b59b84245249cd24bec7205069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.257804 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6229e4efd84e43a77cf359abdaa648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.263288 6251 raft_consensus.cc:491] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.263541 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e065ae748a41d68ad1abe0471b5ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.263895 5597 raft_consensus.cc:357] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.263660 6251 raft_consensus.cc:513] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.264487 5597 raft_consensus.cc:383] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:17.264678 5597 raft_consensus.cc:738] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:17.265203 5597 consensus_queue.cc:260] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.267436 5773 raft_consensus.cc:2267] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:17.279382 5398 consensus_peers.cc:487] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 491626848ad54be3a18186c65d6dbb60. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:17.279316 5299 tablet_service.cc:1467] Processing CreateTablet for tablet 26e45db2f7db4c74bab4367857412ec5 (DEFAULT_TABLE table=default.Table_7d46d595a1834776a8052c7e8c75ada2 [id=7ca49fbdd182424eb17be3d39490d37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.279528 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6229e4efd84e43a77cf359abdaa648 (DEFAULT_TABLE table=default.Table_e73d9dd6e1d44352936cc594ddd9de99 [id=61645b59b84245249cd24bec7205069b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:17.279929 5398 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:17.280211 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e45db2f7db4c74bab4367857412ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.249387 6252 raft_consensus.cc:491] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.226608 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 191) took 2013 ms. Trace:
I20250110 20:57:17.280789 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6229e4efd84e43a77cf359abdaa648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.282187 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 16aaabfff9a04310b19a3195d0b41d7a (DEFAULT_TABLE table=default.Table_461c0135de1f4d318a4f6ceaf204ae45 [id=aa4e3169f5294e5e8ee337f07b08e422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.282318 5437 tablet_service.cc:1467] Processing CreateTablet for tablet c7961f8bd3ff42ffab214a04c704fd26 (DEFAULT_TABLE table=default.Table_1709c1058a134005a1475580de97779b [id=cf5d35e4dfbf45688b9ba58240ac8262]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.283303 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16aaabfff9a04310b19a3195d0b41d7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.283805 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 79e065ae748a41d68ad1abe0471b5ed9 (DEFAULT_TABLE table=default.Table_667604fe371a4f43a1c96e082923f62a [id=19e4e1b23fd74c278cad78decb74aaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.284652 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79e065ae748a41d68ad1abe0471b5ed9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.285782 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7961f8bd3ff42ffab214a04c704fd26. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.300217 5595 ts_tablet_manager.cc:1918] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:17.302939 5594 tablet_bootstrap.cc:654] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:17.303639 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet c7961f8bd3ff42ffab214a04c704fd26
I20250110 20:57:17.303694 5595 log.cc:1198] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/7462d21b91c2441fb9e11383466e9ca7
I20250110 20:57:17.305047 5595 ts_tablet_manager.cc:1939] T 7462d21b91c2441fb9e11383466e9ca7 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:17.307281 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 190) took 1401 ms. Trace:
I20250110 20:57:17.309623 5595 rpcz_store.cc:276] 0110 20:57:15.906616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.910318 (+ 3702us) service_pool.cc:225] Handling call
0110 20:57:17.183862 (+1273544us) ts_tablet_manager.cc:1145] Deleting tablet 7462d21b91c2441fb9e11383466e9ca7
0110 20:57:17.183907 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.213952 (+ 30045us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.300025 (+ 86073us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.307219 (+ 7194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1272582}
I20250110 20:57:17.311605 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 191) took 1381 ms. Trace:
I20250110 20:57:17.312702 5595 rpcz_store.cc:276] 0110 20:57:15.930125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.930241 (+ 116us) service_pool.cc:225] Handling call
0110 20:57:17.311307 (+1381066us) ts_tablet_manager.cc:1145] Deleting tablet c57941aee12a4ebea2b47f4727fc6eee
0110 20:57:17.311343 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.311558 (+ 215us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1380469}
I20250110 20:57:17.314352 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 192) took 1372 ms. Trace:
I20250110 20:57:17.254649 6173 raft_consensus.cc:491] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.317591 5595 rpcz_store.cc:276] 0110 20:57:15.942313 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.943187 (+ 874us) service_pool.cc:225] Handling call
0110 20:57:17.314068 (+1370881us) ts_tablet_manager.cc:1145] Deleting tablet bae52ea080f54d228b6b76922248ed5a
0110 20:57:17.314102 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.314308 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1365495}
I20250110 20:57:17.267524 5597 ts_tablet_manager.cc:1428] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20250110 20:57:17.329264 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 26e45db2f7db4c74bab4367857412ec5 (DEFAULT_TABLE table=default.Table_7d46d595a1834776a8052c7e8c75ada2 [id=7ca49fbdd182424eb17be3d39490d37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.330195 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e45db2f7db4c74bab4367857412ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.331166 4899 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:17.331887 4899 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:17.332509 4899 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (table default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145]). Expected version 1 got 0
I20250110 20:57:17.333251 4899 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:17.334407 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_b26866e23984441f96358c0d662acea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:17.335819 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b26866e23984441f96358c0d662acea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.337013 5597 tablet_bootstrap.cc:492] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:17.338747 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet c7961f8bd3ff42ffab214a04c704fd26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:17.335353 6173 raft_consensus.cc:513] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.343395 5155 tablet_service.cc:1467] Processing CreateTablet for tablet 5feb54cd239b4a80894d45898776fce5 (DEFAULT_TABLE table=default.Table_a0d2f2c0efd3421980eae4849cea168a [id=745914d213dd4df694528230b1ac4e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.344260 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5feb54cd239b4a80894d45898776fce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.346418 6253 raft_consensus.cc:695] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:17.356133 5773 ts_tablet_manager.cc:1905] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.355654 6253 consensus_queue.cc:237] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:17.357611 6259 log.cc:926] Time spent T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20250110 20:57:17.357714 6173 leader_election.cc:290] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:17.359082 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet c7961f8bd3ff42ffab214a04c704fd26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:17.359226 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]) successfully deleted
W20250110 20:57:17.360157 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet c57941aee12a4ebea2b47f4727fc6eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c57941aee12a4ebea2b47f4727fc6eee already in progress: creating tablet
I20250110 20:57:17.360112 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 4aaa42bc5ab042e79baee1ea838851eb (DEFAULT_TABLE table=default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:17.361598 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet bae52ea080f54d228b6b76922248ed5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bae52ea080f54d228b6b76922248ed5a already in progress: creating tablet
I20250110 20:57:17.363129 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 4aaa42bc5ab042e79baee1ea838851eb (DEFAULT_TABLE table=default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.364035 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aaa42bc5ab042e79baee1ea838851eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.369128 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d390bf037cd241728651d40178602a0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:17.372167 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a2c1f3f4459a412885b0f548c3b71361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:17.373515 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2c1f3f4459a412885b0f548c3b71361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:17.376762 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d390bf037cd241728651d40178602a0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.306180 5592 rpcz_store.cc:276] 0110 20:57:15.213201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.214869 (+ 1668us) service_pool.cc:225] Handling call
0110 20:57:16.980038 (+1765169us) ts_tablet_manager.cc:1145] Deleting tablet 7cf0c87159574bfcbc7a34abe508f9d6
0110 20:57:16.980104 (+ 66us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.082843 (+102739us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.182839 (+ 99996us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.226237 (+ 43398us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51736,"tablet-delete.queue_time_us":1753926}
I20250110 20:57:17.395548 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 184) took 2228 ms. Trace:
I20250110 20:57:17.395823 5592 rpcz_store.cc:276] 0110 20:57:15.167162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.208469 (+ 41307us) service_pool.cc:225] Handling call
0110 20:57:17.395322 (+2186853us) ts_tablet_manager.cc:1145] Deleting tablet ae5956b1aa17449497f9695a111ebb66
0110 20:57:17.395359 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.395500 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2125089}
I20250110 20:57:17.396667 5592 tablet_replica.cc:331] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:17.397325 5592 raft_consensus.cc:2238] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:17.397943 5592 raft_consensus.cc:2267] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.399446 5592 ts_tablet_manager.cc:1905] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.400844 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527bbddda8184a6eb67c2411ae50db63" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:17.401389 5460 raft_consensus.cc:2463] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:17.402381 5110 leader_election.cc:304] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:17.404374 6250 raft_consensus.cc:491] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.404817 6250 raft_consensus.cc:513] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.406553 6250 leader_election.cc:290] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:17.314788 6251 leader_election.cc:290] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:17.316458 5441 tablet_service.cc:1467] Processing CreateTablet for tablet ba0d83a50ec0479f9673a57370b0e879 (DEFAULT_TABLE table=default.Table_682824c261c6472dbff5377598f241af [id=079e1a9cdfa24770af8a27d3870a24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.407791 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba0d83a50ec0479f9673a57370b0e879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.316640 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 9f6229e4efd84e43a77cf359abdaa648 (DEFAULT_TABLE table=default.Table_e73d9dd6e1d44352936cc594ddd9de99 [id=61645b59b84245249cd24bec7205069b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.409057 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f6229e4efd84e43a77cf359abdaa648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.317174 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet c7961f8bd3ff42ffab214a04c704fd26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:17.341534 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6914e258314423c859eb31e85853274" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:17.410773 5463 raft_consensus.cc:2463] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:17.414211 5110 leader_election.cc:304] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:17.387800 5399 leader_election.cc:304] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
W20250110 20:57:17.415292 5399 raft_consensus.cc:2681] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250110 20:57:17.389122 5398 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 108675us
W20250110 20:57:17.421615 5398 leader_election.cc:343] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 527bbddda8184a6eb67c2411ae50db63
I20250110 20:57:17.425735 5592 ts_tablet_manager.cc:1918] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:17.426126 5592 log.cc:1198] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:17.390858 5439 tablet_service.cc:1514] Processing DeleteTablet for tablet 01962c5b7ae64f80b998948d18250b4c with delete_type TABLET_DATA_DELETED (replaced by tablet b5becf70b58847559b70ce27edc30398) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:17.426748 5592 ts_tablet_manager.cc:1939] T ba1b57de9e4b4b2ab6c3bce8fe78166a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:17.428222 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 194) took 1058 ms. Trace:
I20250110 20:57:17.428509 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 5feb54cd239b4a80894d45898776fce5 (DEFAULT_TABLE table=default.Table_a0d2f2c0efd3421980eae4849cea168a [id=745914d213dd4df694528230b1ac4e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.428555 5595 rpcz_store.cc:276] 0110 20:57:16.369827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.370206 (+ 379us) service_pool.cc:225] Handling call
0110 20:57:17.318995 (+948789us) ts_tablet_manager.cc:1145] Deleting tablet d310cf52aa8c4930a7821117a46dc958
0110 20:57:17.319028 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.428168 (+109140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":934431}
I20250110 20:57:17.429487 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5feb54cd239b4a80894d45898776fce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.429559 5595 tablet_replica.cc:331] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:17.430265 5595 raft_consensus.cc:2238] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:17.433828 5595 raft_consensus.cc:2267] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.435329 5595 ts_tablet_manager.cc:1905] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.334730 4899 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]). Expected version 1 got 0
I20250110 20:57:17.441349 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 5feb54cd239b4a80894d45898776fce5 (DEFAULT_TABLE table=default.Table_a0d2f2c0efd3421980eae4849cea168a [id=745914d213dd4df694528230b1ac4e41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.442332 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5feb54cd239b4a80894d45898776fce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.443310 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 186) took 2275 ms. Trace:
I20250110 20:57:17.443598 5592 rpcz_store.cc:276] 0110 20:57:15.167995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.170241 (+ 2246us) service_pool.cc:225] Handling call
0110 20:57:17.396542 (+2226301us) ts_tablet_manager.cc:1145] Deleting tablet ba1b57de9e4b4b2ab6c3bce8fe78166a
0110 20:57:17.396577 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.407444 (+ 10867us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.417396 (+ 9952us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.443258 (+ 25862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2125644}
I20250110 20:57:17.444640 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 185) took 2277 ms. Trace:
I20250110 20:57:17.441349 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 26e45db2f7db4c74bab4367857412ec5 (DEFAULT_TABLE table=default.Table_7d46d595a1834776a8052c7e8c75ada2 [id=7ca49fbdd182424eb17be3d39490d37e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.447451 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e45db2f7db4c74bab4367857412ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.440320 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 7cf0c87159574bfcbc7a34abe508f9d6 (table default.Table_fc5f72ec93dd40bbaeb1be5da336b098 [id=6cc26e56afd04d78abff401b7ae32d99]) successfully deleted
W20250110 20:57:17.452076 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet ae5956b1aa17449497f9695a111ebb66 because the tablet was not found. No further retry: Not found: Tablet not found: ae5956b1aa17449497f9695a111ebb66
I20250110 20:57:17.437378 4899 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_a9b22870bca84943b105815613f88adb [id=2c78ee52e44a46fe877f474a7ecb771b]). Expected version 1 got 0
I20250110 20:57:17.444900 5592 rpcz_store.cc:276] 0110 20:57:15.167719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.170019 (+ 2300us) service_pool.cc:225] Handling call
0110 20:57:17.444428 (+2274409us) ts_tablet_manager.cc:1145] Deleting tablet 31b65488c884452d876ced1c3d1efb9a
0110 20:57:17.444465 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.444596 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173100}
I20250110 20:57:17.454609 5592 tablet_replica.cc:331] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:17.455724 5592 raft_consensus.cc:2238] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:17.453716 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet ba1b57de9e4b4b2ab6c3bce8fe78166a (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
W20250110 20:57:17.464866 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 31b65488c884452d876ced1c3d1efb9a because the tablet was not found. No further retry: Not found: Tablet not found: 31b65488c884452d876ced1c3d1efb9a
I20250110 20:57:17.464509 4899 catalog_manager.cc:5526] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
W20250110 20:57:17.457798 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet d310cf52aa8c4930a7821117a46dc958 because the tablet was not found. No further retry: Not found: Tablet not found: d310cf52aa8c4930a7821117a46dc958
W20250110 20:57:17.468691 5111 leader_election.cc:343] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 527bbddda8184a6eb67c2411ae50db63
I20250110 20:57:17.472347 5600 tablet_bootstrap.cc:492] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:17.472816 5600 ts_tablet_manager.cc:1397] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.425s user 0.009s sys 0.004s
I20250110 20:57:17.465988 6173 raft_consensus.cc:2798] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:17.438143 6059 raft_consensus.cc:2798] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:17.476368 5600 raft_consensus.cc:357] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.483645 5773 ts_tablet_manager.cc:1918] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:17.477222 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:17.484071 5773 log.cc:1198] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:17.485908 5773 ts_tablet_manager.cc:1939] T f0b5ba9f38c1493bba01180744f080e1 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:17.467944 5594 tablet_bootstrap.cc:492] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:17.496358 5592 raft_consensus.cc:2267] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.493144 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8713c1e5a743a489de11bc65857bff (DEFAULT_TABLE table=default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.496171 5594 ts_tablet_manager.cc:1397] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.346s user 0.004s sys 0.005s
I20250110 20:57:17.498351 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8713c1e5a743a489de11bc65857bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.499104 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527bbddda8184a6eb67c2411ae50db63" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:17.500710 6256 raft_consensus.cc:491] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.490016 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 4aaa42bc5ab042e79baee1ea838851eb (DEFAULT_TABLE table=default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.501116 6256 raft_consensus.cc:513] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.502130 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aaa42bc5ab042e79baee1ea838851eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.502904 6256 leader_election.cc:290] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:17.470036 5176 raft_consensus.cc:3054] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.505879 5592 ts_tablet_manager.cc:1905] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.501310 5594 raft_consensus.cc:357] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.509181 5594 raft_consensus.cc:383] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:17.481310 5294 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8713c1e5a743a489de11bc65857bff (DEFAULT_TABLE table=default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.510273 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8713c1e5a743a489de11bc65857bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.477515 6059 raft_consensus.cc:491] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:17.513314 6059 raft_consensus.cc:3054] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:17.516041 5399 leader_election.cc:343] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:17.516757 5314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
W20250110 20:57:17.516924 5399 consensus_peers.cc:487] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:17.517323 5314 raft_consensus.cc:2463] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:17.517535 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:17.335239 6252 raft_consensus.cc:513] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.518417 6251 consensus_queue.cc:1035] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:17.519779 6252 leader_election.cc:290] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:17.477440 6173 raft_consensus.cc:491] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:17.520346 6173 raft_consensus.cc:3054] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.482318 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb (DEFAULT_TABLE table=default.Table_f70bf3182e2a4bcbafbc98e2977b4f2d [id=4bc4846633fc4e74b4d2e79c01e3db99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.521862 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.526476 5398 leader_election.cc:304] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:17.526907 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aaa42bc5ab042e79baee1ea838851eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.527437 6205 raft_consensus.cc:2798] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:17.527734 6205 raft_consensus.cc:491] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:17.528020 6205 raft_consensus.cc:3054] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.530064 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637a39e4264b40dd9480dc13e37359d9" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:17.530737 5600 raft_consensus.cc:383] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:17.531019 5600 raft_consensus.cc:738] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:17.531565 5600 consensus_queue.cc:260] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.533969 6205 raft_consensus.cc:513] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.535423 4892 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet fde470279464496aa7efdee3800754f3
I20250110 20:57:17.536501 6205 leader_election.cc:290] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:17.536747 4892 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 236) took 1365 ms. Trace:
I20250110 20:57:17.538501 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet fde470279464496aa7efdee3800754f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:17.538571 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet fde470279464496aa7efdee3800754f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:17.463689 5509 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
I20250110 20:57:17.537034 4892 rpcz_store.cc:276] 0110 20:57:16.171170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.171271 (+ 101us) service_pool.cc:225] Handling call
0110 20:57:16.201377 (+ 30106us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:16.201912 (+ 535us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:16.202032 (+ 120us) catalog_manager.cc:2762] Locking tablets
0110 20:57:16.202552 (+ 520us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:16.426035 (+223483us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:16.426230 (+ 195us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:17.535342 (+1109112us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:17.536698 (+ 1356us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:16.210427 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:16.210730 (+ 303us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:16.210749 (+ 19us) write_op.cc:435] Acquired schema lock
0110 20:57:16.210761 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:16.211051 (+ 290us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:16.211084 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:16.211105 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:16.211343 (+ 238us) tablet.cc:698] Row locks acquired
0110 20:57:16.211356 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:16.211451 (+ 95us) write_op.cc:270] Start()
0110 20:57:16.211524 (+ 73us) write_op.cc:276] Timestamp: P: 1736542636211408 usec, L: 0
0110 20:57:16.211538 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:16.211948 (+ 410us) log.cc:844] Serialized 796 byte log entry
0110 20:57:16.418383 (+206435us) op_driver.cc:464] REPLICATION: finished
0110 20:57:16.424652 (+ 6269us) write_op.cc:301] APPLY: starting
0110 20:57:16.424706 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:16.424800 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:16.424890 (+ 90us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:16.425147 (+ 257us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:16.425268 (+ 121us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:16.425304 (+ 36us) write_op.cc:312] APPLY: finished
0110 20:57:16.425490 (+ 186us) log.cc:844] Serialized 29 byte log entry
0110 20:57:16.425624 (+ 134us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:16.425684 (+ 60us) write_op.cc:454] Released schema lock
0110 20:57:16.425714 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:16.425895 (+ 181us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:16.425923 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":6117,"apply.run_cpu_time_us":1222,"apply.run_wall_time_us":1225,"num_ops":2,"prepare.queue_time_us":6210,"prepare.run_cpu_time_us":1693,"prepare.run_wall_time_us":2625,"replication_time_us":206753}]]}
W20250110 20:57:17.540220 5398 consensus_peers.cc:487] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7462d21b91c2441fb9e11383466e9ca7. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:17.540196 4892 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e412e39463cf4fc0bf8a55ba265ffb25" } modify_external_catalogs: true
I20250110 20:57:17.540649 4892 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e412e39463cf4fc0bf8a55ba265ffb25" } modify_external_catalogs: true
W20250110 20:57:17.541505 5398 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 637a39e4264b40dd9480dc13e37359d9
I20250110 20:57:17.542433 6251 consensus_queue.cc:1035] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:17.546861 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:17.548978 5110 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
W20250110 20:57:17.554744 5110 consensus_peers.cc:487] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:17.545222 5399 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:17.558578 5399 leader_election.cc:304] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:17.559214 6205 raft_consensus.cc:2743] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:17.559727 5399 leader_election.cc:343] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:17.566601 5301 tablet_service.cc:1467] Processing CreateTablet for tablet adfed905cadb4e82acd3ff066c8c4cba (DEFAULT_TABLE table=default.Table_68999c6dd9a84b24b631302d58de6c02 [id=f7a83368bbd149f4839309beb9f7baad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.568783 5597 tablet_bootstrap.cc:654] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:17.569552 5257 consensus_peers.cc:487] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:17.570991 5314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9d0249798e4f24adf0b36d9e059b7e" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:17.578717 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8e3dc8862ef2488b9fe7b37dfc1f3a09" } new_table_name: "default.Table_82e8a56b6f35429a9996b1622048bf8f" modify_external_catalogs: true
I20250110 20:57:17.582046 4897 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_d1c3754a45fb4d75b94dffa666cfc9a8 [id=0f7debac7d1e4de694325d88c01a5f76] alter complete (version 2)
I20250110 20:57:17.582135 5594 raft_consensus.cc:738] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:17.582734 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_8fefcbeeff174088ad6b60ddc002689d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:17.582784 5594 consensus_queue.cc:260] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:17.584059 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fefcbeeff174088ad6b60ddc002689d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.584831 5594 ts_tablet_manager.cc:1428] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.086s user 0.004s sys 0.000s
I20250110 20:57:17.589062 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_66d144e5d786435c8a4a7a14638914ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:17.590371 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66d144e5d786435c8a4a7a14638914ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.600517 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb (DEFAULT_TABLE table=default.Table_f70bf3182e2a4bcbafbc98e2977b4f2d [id=4bc4846633fc4e74b4d2e79c01e3db99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.601526 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.605950 5438 tablet_service.cc:1467] Processing CreateTablet for tablet adfed905cadb4e82acd3ff066c8c4cba (DEFAULT_TABLE table=default.Table_68999c6dd9a84b24b631302d58de6c02 [id=f7a83368bbd149f4839309beb9f7baad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.606856 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adfed905cadb4e82acd3ff066c8c4cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.605819 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 205) took 1895 ms. Trace:
W20250110 20:57:17.608080 4897 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 42) took 1210 ms (client timeout 1000 ms). Trace:
I20250110 20:57:17.608078 5773 rpcz_store.cc:276] 0110 20:57:15.710056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.769510 (+ 59454us) service_pool.cc:225] Handling call
0110 20:57:17.245265 (+1475755us) ts_tablet_manager.cc:1145] Deleting tablet f0b5ba9f38c1493bba01180744f080e1
0110 20:57:17.245306 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.372089 (+126783us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.483411 (+111322us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.605766 (+122355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474987}
I20250110 20:57:17.609050 5773 tablet_replica.cc:331] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:17.609720 5773 raft_consensus.cc:2238] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
W20250110 20:57:17.608368 4897 rpcz_store.cc:269] 0110 20:57:16.397361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.404668 (+ 7307us) service_pool.cc:225] Handling call
0110 20:57:16.445019 (+ 40351us) spinlock_profiling.cc:243] Waited 25.4 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:17.608014 (+1162995us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:17.538196 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:17.547726 (+ 9530us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:17.547749 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:17.547761 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:17.547997 (+ 236us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:17.548034 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:17.548054 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:17.548182 (+ 128us) tablet.cc:698] Row locks acquired
0110 20:57:17.548195 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:17.548301 (+ 106us) write_op.cc:270] Start()
0110 20:57:17.548375 (+ 74us) write_op.cc:276] Timestamp: P: 1736542637548219 usec, L: 0
0110 20:57:17.548389 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:17.561593 (+ 13204us) log.cc:844] Serialized 356 byte log entry
0110 20:57:17.579544 (+ 17951us) op_driver.cc:464] REPLICATION: finished
0110 20:57:17.580503 (+ 959us) write_op.cc:301] APPLY: starting
0110 20:57:17.580557 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:17.580636 (+ 79us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:17.580735 (+ 99us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:17.580867 (+ 132us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:17.580984 (+ 117us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:17.581023 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:57:17.581172 (+ 149us) log.cc:844] Serialized 23 byte log entry
0110 20:57:17.581322 (+ 150us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.581378 (+ 56us) write_op.cc:454] Released schema lock
0110 20:57:17.581423 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:17.581618 (+ 195us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.581639 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":21948,"reported_tablets":12,"spinlock_wait_cycles":58514304,"child_traces":[["op",{"apply.queue_time_us":814,"apply.run_cpu_time_us":1087,"apply.run_wall_time_us":1088,"num_ops":1,"prepare.queue_time_us":3603,"prepare.run_cpu_time_us":1833,"prepare.run_wall_time_us":24677,"raft.queue_time_us":83,"raft.run_cpu_time_us":587,"raft.run_wall_time_us":587,"replication_time_us":31070}]]}
I20250110 20:57:17.610597 5111 leader_election.cc:304] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:17.624070 6059 raft_consensus.cc:513] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:17.638761 5600 ts_tablet_manager.cc:1428] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.164s user 0.004s sys 0.002s
I20250110 20:57:17.647207 5296 tablet_service.cc:1467] Processing CreateTablet for tablet d099f846d1f94e07ac478d904451953d (DEFAULT_TABLE table=default.Table_1d59ab5eaa3845d4a405718a816829cb [id=d3f5513ed82b45aeb10bfd3e7c181ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.648276 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adfed905cadb4e82acd3ff066c8c4cba. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:17.650246 4894 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d9672dc9cbb4e27bf6b01f513b6ccb8 has no consensus state
I20250110 20:57:17.656026 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d099f846d1f94e07ac478d904451953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.659272 5595 ts_tablet_manager.cc:1918] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:17.659596 5595 log.cc:1198] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/c44ca47c747544b6afec83b399dcdcb5
I20250110 20:57:17.660135 5595 ts_tablet_manager.cc:1939] T c44ca47c747544b6afec83b399dcdcb5 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:17.660760 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 195) took 1246 ms. Trace:
I20250110 20:57:17.663154 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:17.663682 5319 raft_consensus.cc:3054] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.673713 5600 tablet_bootstrap.cc:492] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:17.678120 5594 tablet_bootstrap.cc:492] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:17.689316 5600 tablet_bootstrap.cc:654] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:17.697544 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 87016.3us
W20250110 20:57:17.698351 5111 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:17.698668 5111 leader_election.cc:304] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:17.706041 6175 consensus_queue.cc:1035] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:57:17.711314 5594 tablet_bootstrap.cc:654] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:17.661020 5595 rpcz_store.cc:276] 0110 20:57:16.414535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.414798 (+ 263us) service_pool.cc:225] Handling call
0110 20:57:17.429384 (+1014586us) ts_tablet_manager.cc:1145] Deleting tablet c44ca47c747544b6afec83b399dcdcb5
0110 20:57:17.429448 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.450237 (+ 20789us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.659087 (+208850us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.660714 (+ 1627us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3124,"tablet-delete.queue_time_us":1014055}
I20250110 20:57:17.714602 5595 tablet_replica.cc:331] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:17.715359 5595 raft_consensus.cc:2238] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:17.715329 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet c44ca47c747544b6afec83b399dcdcb5 (table default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]) successfully deleted
I20250110 20:57:17.715874 5595 raft_consensus.cc:2267] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.715029 5319 raft_consensus.cc:2463] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:17.718499 5438 tablet_service.cc:1467] Processing CreateTablet for tablet d099f846d1f94e07ac478d904451953d (DEFAULT_TABLE table=default.Table_1d59ab5eaa3845d4a405718a816829cb [id=d3f5513ed82b45aeb10bfd3e7c181ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.726080 5600 tablet_bootstrap.cc:492] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:17.727571 5600 ts_tablet_manager.cc:1397] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.003s
I20250110 20:57:17.719941 5398 leader_election.cc:304] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:17.730993 5592 ts_tablet_manager.cc:1918] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:17.731091 6205 raft_consensus.cc:2798] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:17.731763 6205 raft_consensus.cc:695] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:17.732504 6205 consensus_queue.cc:237] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.740394 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d099f846d1f94e07ac478d904451953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.741272 5595 ts_tablet_manager.cc:1905] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.739992 5600 raft_consensus.cc:357] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.742642 5600 raft_consensus.cc:383] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:17.743621 5600 raft_consensus.cc:738] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:17.748860 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_3f8bafa0c19642adb6062c8156965b27" } modify_external_catalogs: true
I20250110 20:57:17.749346 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_3f8bafa0c19642adb6062c8156965b27" } modify_external_catalogs: true
I20250110 20:57:17.756666 4897 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a2c1f3f4459a412885b0f548c3b71361" } new_table_name: "default.Table_53bbb78e4859491ba4013f5dd868d449" modify_external_catalogs: true
I20250110 20:57:17.762837 6172 raft_consensus.cc:2743] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:17.763796 5773 raft_consensus.cc:2267] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.747721 5600 consensus_queue.cc:260] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.770753 5773 ts_tablet_manager.cc:1905] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.777009 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 75862d0008d44a6c8694aa02dae45a80 (DEFAULT_TABLE table=default.Table_46ff9199b08c4e779e9e21081434e52e [id=a56d228fbebe4e6b8501fa526d10eb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.757230 5594 tablet_bootstrap.cc:492] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:17.778096 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75862d0008d44a6c8694aa02dae45a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.778429 5594 ts_tablet_manager.cc:1397] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.002s
I20250110 20:57:17.731424 5592 log.cc:1198] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/e361fb8b47d74548b5a2326d25a9e28c
I20250110 20:57:17.781910 5592 ts_tablet_manager.cc:1939] T e361fb8b47d74548b5a2326d25a9e28c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:17.782675 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 182) took 2657 ms. Trace:
I20250110 20:57:17.783017 5592 rpcz_store.cc:276] 0110 20:57:15.124880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.125419 (+ 539us) service_pool.cc:225] Handling call
0110 20:57:17.454472 (+2329053us) ts_tablet_manager.cc:1145] Deleting tablet e361fb8b47d74548b5a2326d25a9e28c
0110 20:57:17.454512 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.513056 (+ 58544us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.730755 (+217699us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.782623 (+ 51868us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183009}
I20250110 20:57:17.784231 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet e361fb8b47d74548b5a2326d25a9e28c (table default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]) successfully deleted
I20250110 20:57:17.784848 5592 tablet_replica.cc:331] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:17.785662 5592 raft_consensus.cc:2238] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:17.786175 5592 raft_consensus.cc:2267] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:17.786361 6277 tablet.cc:1721] T cb7f09df8e274f3e9542292d5c1ec6c6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:17.787976 5592 ts_tablet_manager.cc:1905] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:17.790616 5600 ts_tablet_manager.cc:1428] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.062s user 0.000s sys 0.008s
I20250110 20:57:17.791409 5600 tablet_bootstrap.cc:492] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:17.791352 6279 tablet.cc:1721] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:17.791683 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 75862d0008d44a6c8694aa02dae45a80 (DEFAULT_TABLE table=default.Table_46ff9199b08c4e779e9e21081434e52e [id=a56d228fbebe4e6b8501fa526d10eb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:17.792440 6258 raft_consensus.cc:2743] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:17.792697 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75862d0008d44a6c8694aa02dae45a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.781061 5594 raft_consensus.cc:357] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.800928 5594 raft_consensus.cc:383] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:17.801187 5594 raft_consensus.cc:738] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:17.801806 5594 consensus_queue.cc:260] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.801702 6276 tablet.cc:1721] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:17.804046 5594 ts_tablet_manager.cc:1428] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.025s user 0.002s sys 0.003s
I20250110 20:57:17.804806 5594 tablet_bootstrap.cc:492] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:17.815438 5600 tablet_bootstrap.cc:654] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:17.822739 6278 tablet.cc:1721] T cb7f09df8e274f3e9542292d5c1ec6c6 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:17.830199 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_c1d432de9a1f4c218e0150fee53f6e6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:17.831646 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1d432de9a1f4c218e0150fee53f6e6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.834733 6059 leader_election.cc:290] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:17.838670 5600 tablet_bootstrap.cc:492] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:17.839143 5600 ts_tablet_manager.cc:1397] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.005s
I20250110 20:57:17.842895 5600 raft_consensus.cc:357] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.862171 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6914e258314423c859eb31e85853274" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:17.862818 5460 raft_consensus.cc:3054] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:17.899724 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d569e0fc30134336bc26374a5eaeb7cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:17.901064 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d569e0fc30134336bc26374a5eaeb7cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.883764 6175 raft_consensus.cc:491] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:17.913761 6175 raft_consensus.cc:513] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.915637 6175 leader_election.cc:290] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:17.916883 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 32cf221df30e4e78bd2a8078c26d29ba
I20250110 20:57:17.917573 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3104928188c489d93bfd499d148fa5d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:17.919452 4897 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51fae5f59d4642a09acee5a687b91717 has no consensus state
W20250110 20:57:17.919852 5255 leader_election.cc:343] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:17.926187 5600 raft_consensus.cc:383] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:17.927445 5600 raft_consensus.cc:738] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:17.926620 5460 raft_consensus.cc:2463] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:17.929177 5600 consensus_queue.cc:260] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:17.933743 5292 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cf221df30e4e78bd2a8078c26d29ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:17.946344 5595 ts_tablet_manager.cc:1918] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:17.936790 4891 catalog_manager.cc:5526] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194). New cstate: current_term: 1 leader_uuid: "46005a5920cc4a3697a5c61b1c271384" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:17.949810 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (table default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145]). Expected version 1 got 0
I20250110 20:57:17.954259 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]). Expected version 1 got 0
I20250110 20:57:17.959201 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:17.961556 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_1588a911bebe4556aee77ec7ebb160cf [id=2c78ee52e44a46fe877f474a7ecb771b]). Expected version 2 got 0
I20250110 20:57:17.967646 5595 log.cc:1198] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/e508061e93cc430cb88f6d10d9684f00
I20250110 20:57:17.957928 5368 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:57:17.974207 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cf221df30e4e78bd2a8078c26d29ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:17.968834 5595 ts_tablet_manager.cc:1939] T e508061e93cc430cb88f6d10d9684f00 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:17.975212 5600 ts_tablet_manager.cc:1428] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.136s user 0.007s sys 0.000s
I20250110 20:57:17.981665 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 196) took 1489 ms. Trace:
I20250110 20:57:17.986639 5600 tablet_bootstrap.cc:492] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:17.985628 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_c8c39c1a1dde4e46a4f0aaabe3045c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:17.989681 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
W20250110 20:57:17.990211 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8c39c1a1dde4e46a4f0aaabe3045c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:17.990829 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:17.991448 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:17.991889 4891 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff]). Expected version 1 got 0
I20250110 20:57:17.986897 4896 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_46ff9199b08c4e779e9e21081434e52e" } new_table_name: "default.Table_15deb2e69b6f44509175a68074b279df" modify_external_catalogs: true
I20250110 20:57:17.982280 5595 rpcz_store.cc:276] 0110 20:57:16.492590 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.492712 (+ 122us) service_pool.cc:225] Handling call
0110 20:57:17.714457 (+1221745us) ts_tablet_manager.cc:1145] Deleting tablet e508061e93cc430cb88f6d10d9684f00
0110 20:57:17.714498 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.772408 (+ 57910us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.946104 (+173696us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:17.981578 (+ 35474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1221229}
I20250110 20:57:17.997712 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet e508061e93cc430cb88f6d10d9684f00 (table default.Table_01d2ed1a594948c4995e0b934e4d2860 [id=e9f106d1b737444baaea6e97a8a3884a]) successfully deleted
I20250110 20:57:18.003512 5594 tablet_bootstrap.cc:654] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.007301 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 197) took 1121 ms. Trace:
I20250110 20:57:17.959813 5110 leader_election.cc:304] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:17.866987 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 7f8713c1e5a743a489de11bc65857bff (DEFAULT_TABLE table=default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.007864 5595 rpcz_store.cc:276] 0110 20:57:16.885398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.885616 (+ 218us) service_pool.cc:225] Handling call
0110 20:57:17.997536 (+1111920us) ts_tablet_manager.cc:1145] Deleting tablet a95f05fc5afe4877a606b501dbd09551
0110 20:57:17.997608 (+ 72us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.007171 (+ 9563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1111326}
I20250110 20:57:18.017972 5595 tablet_replica.cc:331] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:18.000437 5145 tablet_service.cc:1467] Processing CreateTablet for tablet adfed905cadb4e82acd3ff066c8c4cba (DEFAULT_TABLE table=default.Table_68999c6dd9a84b24b631302d58de6c02 [id=f7a83368bbd149f4839309beb9f7baad]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:18.018188 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet a95f05fc5afe4877a606b501dbd09551 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a95f05fc5afe4877a606b501dbd09551 already in progress: creating tablet
I20250110 20:57:18.022401 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet f0b5ba9f38c1493bba01180744f080e1 (table default.Table_5356ef036bd048488ffadd8d4b4caf58 [id=cdc76f7bfc594bb8bc9f56ccca1a2155]) successfully deleted
I20250110 20:57:17.833491 6173 raft_consensus.cc:513] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.026481 5595 raft_consensus.cc:2238] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:18.027287 5595 raft_consensus.cc:2267] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.028978 5595 ts_tablet_manager.cc:1905] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:18.013137 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 53412.4us
I20250110 20:57:17.831655 5174 raft_consensus.cc:3054] T bd9d0249798e4f24adf0b36d9e059b7e P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:18.019366 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac07ac6fc9f477ba7f9accc5533be39 with delete_type TABLET_DATA_DELETED (replaced by tablet 371fb7408f0149c8b00659e09ebaf48a) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.031543 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet cd0a666892d04e97a4b9967e38ab0946 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.013507 5154 tablet_service.cc:1514] Processing DeleteTablet for tablet fde470279464496aa7efdee3800754f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.019706 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f8713c1e5a743a489de11bc65857bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.019870 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet 527bbddda8184a6eb67c2411ae50db63 with delete_type TABLET_DATA_DELETED (replaced by tablet 9633eb4580c04f11a48ff5055e257c50) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.020153 5147 tablet_service.cc:1514] Processing DeleteTablet for tablet 491626848ad54be3a18186c65d6dbb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.021502 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cf221df30e4e78bd2a8078c26d29ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.040426 5600 tablet_bootstrap.cc:654] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:17.870095 6213 raft_consensus.cc:357] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 1 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:18.043534 5592 ts_tablet_manager.cc:1918] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:18.044289 5592 log.cc:1198] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:18.022018 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 75862d0008d44a6c8694aa02dae45a80 (DEFAULT_TABLE table=default.Table_46ff9199b08c4e779e9e21081434e52e [id=a56d228fbebe4e6b8501fa526d10eb15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.046795 5222 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:57:18.048645 5592 ts_tablet_manager.cc:1939] T f0b5ba9f38c1493bba01180744f080e1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:18.050091 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adfed905cadb4e82acd3ff066c8c4cba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.021255 5156 tablet_service.cc:1467] Processing CreateTablet for tablet d099f846d1f94e07ac478d904451953d (DEFAULT_TABLE table=default.Table_1d59ab5eaa3845d4a405718a816829cb [id=d3f5513ed82b45aeb10bfd3e7c181ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.049906 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3104928188c489d93bfd499d148fa5d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:18.024134 5171 raft_consensus.cc:2388] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in current term 1: Already voted for candidate ac9003be04f340f3886b5625cc928eb0 in this term.
I20250110 20:57:18.057145 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d099f846d1f94e07ac478d904451953d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:17.977850 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb (DEFAULT_TABLE table=default.Table_f70bf3182e2a4bcbafbc98e2977b4f2d [id=4bc4846633fc4e74b4d2e79c01e3db99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.058498 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.061534 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75862d0008d44a6c8694aa02dae45a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.061079 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 188) took 2892 ms. Trace:
I20250110 20:57:18.062112 5592 rpcz_store.cc:276] 0110 20:57:15.168587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.170687 (+ 2100us) service_pool.cc:225] Handling call
0110 20:57:17.784639 (+2613952us) ts_tablet_manager.cc:1145] Deleting tablet f0b5ba9f38c1493bba01180744f080e1
0110 20:57:17.784746 (+ 107us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.904401 (+119655us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.042988 (+138587us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.060922 (+ 17934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2505394}
I20250110 20:57:18.065140 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet f0b5ba9f38c1493bba01180744f080e1 (table default.Table_5356ef036bd048488ffadd8d4b4caf58 [id=cdc76f7bfc594bb8bc9f56ccca1a2155]) successfully deleted
W20250110 20:57:18.065558 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 367194us
I20250110 20:57:18.066895 5592 tablet_replica.cc:331] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:18.071692 5592 raft_consensus.cc:2238] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:18.004995 5773 ts_tablet_manager.cc:1918] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:18.075472 5773 log.cc:1198] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:18.076229 5773 ts_tablet_manager.cc:1939] T 7cf0c87159574bfcbc7a34abe508f9d6 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:18.076514 5600 tablet_bootstrap.cc:492] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:18.077924 5399 leader_election.cc:304] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.078845 6145 raft_consensus.cc:3054] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:18.085074 6145 raft_consensus.cc:2743] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:18.085819 5600 ts_tablet_manager.cc:1397] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.099s user 0.013s sys 0.000s
I20250110 20:57:18.074360 5592 raft_consensus.cc:2267] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.094393 5592 ts_tablet_manager.cc:1905] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.098094 5600 raft_consensus.cc:357] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.098865 5600 raft_consensus.cc:383] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.099364 5600 raft_consensus.cc:738] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.100311 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 206) took 2369 ms. Trace:
I20250110 20:57:18.100623 5773 rpcz_store.cc:276] 0110 20:57:15.710356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.770431 (+ 60075us) service_pool.cc:225] Handling call
0110 20:57:17.608914 (+1838483us) ts_tablet_manager.cc:1145] Deleting tablet 7cf0c87159574bfcbc7a34abe508f9d6
0110 20:57:17.608949 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:17.780947 (+171998us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.004600 (+223653us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.078101 (+ 73501us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":132654,"tablet-delete.queue_time_us":1837871}
I20250110 20:57:18.101864 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 210) took 2047 ms. Trace:
I20250110 20:57:18.100472 5600 consensus_queue.cc:260] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.102146 5773 rpcz_store.cc:276] 0110 20:57:16.054921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.055067 (+ 146us) service_pool.cc:225] Handling call
0110 20:57:18.101554 (+2046487us) ts_tablet_manager.cc:1145] Deleting tablet bae52ea080f54d228b6b76922248ed5a
0110 20:57:18.101593 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.101816 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2045838}
W20250110 20:57:18.105986 5110 consensus_peers.cc:487] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:18.106200 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_31eb7686cc4b4549bab76f9b01c9e311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:18.085510 5509 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
W20250110 20:57:18.107717 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31eb7686cc4b4549bab76f9b01c9e311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:18.109802 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 7cf0c87159574bfcbc7a34abe508f9d6 (table default.Table_fc5f72ec93dd40bbaeb1be5da336b098 [id=6cc26e56afd04d78abff401b7ae32d99]) successfully deleted
W20250110 20:57:18.110388 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet bae52ea080f54d228b6b76922248ed5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bae52ea080f54d228b6b76922248ed5a already in progress: creating tablet
I20250110 20:57:18.110363 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 208) took 2056 ms. Trace:
I20250110 20:57:18.110667 5773 rpcz_store.cc:276] 0110 20:57:16.054363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.062238 (+ 7875us) service_pool.cc:225] Handling call
0110 20:57:18.102865 (+2040627us) ts_tablet_manager.cc:1145] Deleting tablet c57941aee12a4ebea2b47f4727fc6eee
0110 20:57:18.102899 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.110316 (+ 7417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2040108}
I20250110 20:57:18.111527 5773 tablet_replica.cc:331] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:18.112185 5773 raft_consensus.cc:2238] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 20:57:18.112850 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet c57941aee12a4ebea2b47f4727fc6eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c57941aee12a4ebea2b47f4727fc6eee already in progress: creating tablet
I20250110 20:57:18.017805 6059 raft_consensus.cc:2798] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:18.117266 5773 pending_rounds.cc:62] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:18.115638 5597 tablet_bootstrap.cc:492] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:18.118044 5597 ts_tablet_manager.cc:1397] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.781s user 0.016s sys 0.009s
I20250110 20:57:18.126574 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527bbddda8184a6eb67c2411ae50db63" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:18.127302 5600 ts_tablet_manager.cc:1428] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.040s user 0.007s sys 0.000s
I20250110 20:57:18.130268 5600 tablet_bootstrap.cc:492] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:18.137374 4885 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 79e065ae748a41d68ad1abe0471b5ed9 (table default.Table_667604fe371a4f43a1c96e082923f62a [id=19e4e1b23fd74c278cad78decb74aaa5], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 79e065ae748a41d68ad1abe0471b5ed9 consensus state has no leader
W20250110 20:57:18.138131 6290 consensus_queue.cc:1159] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ac9003be04f340f3886b5625cc928eb0" status { last_received { term: 1 index: 2 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 2 } }
I20250110 20:57:18.124711 6059 raft_consensus.cc:695] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:18.146065 6059 consensus_queue.cc:237] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.128648 6173 leader_election.cc:290] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:18.154006 5110 consensus_peers.cc:487] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:18.162099 5600 tablet_bootstrap.cc:654] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.165858 5595 ts_tablet_manager.cc:1918] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:18.166213 5595 log.cc:1198] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/01962c5b7ae64f80b998948d18250b4c
I20250110 20:57:18.166843 5595 ts_tablet_manager.cc:1939] T 01962c5b7ae64f80b998948d18250b4c P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:18.173525 6256 raft_consensus.cc:491] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3c5bff1e70804ab4a1bb63bdb9345ce6)
I20250110 20:57:18.178808 6256 raft_consensus.cc:513] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.179365 6277 tablet.cc:1721] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:18.181046 6256 leader_election.cc:290] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:18.181952 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:18.183108 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_f4762b68ddab45a685e0d06a7703ea3c" } new_table_name: "default.Table_5f3f989b2ecf48a0a72e17cc18a2b095" modify_external_catalogs: true
I20250110 20:57:18.117539 5773 pending_rounds.cc:69] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878641969614848 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "7462d21b91c2441fb9e11383466e9ca7" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_447ad91cdb774363a84b85d71a154011" dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" new_extra_config { } }
I20250110 20:57:18.185341 5773 raft_consensus.cc:2267] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.186972 5594 tablet_bootstrap.cc:492] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:18.187390 5594 ts_tablet_manager.cc:1397] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.383s user 0.005s sys 0.004s
I20250110 20:57:18.190351 5595 tablet_replica.cc:331] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:18.198207 5597 raft_consensus.cc:357] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.204361 5773 ts_tablet_manager.cc:1905] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.198779 5597 raft_consensus.cc:383] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.209192 5597 raft_consensus.cc:738] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.209919 5597 consensus_queue.cc:260] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.211983 5597 ts_tablet_manager.cc:1428] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.094s user 0.005s sys 0.000s
I20250110 20:57:18.212623 5460 raft_consensus.cc:2463] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
W20250110 20:57:18.215454 5399 consensus_peers.cc:487] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:18.220983 5110 leader_election.cc:304] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:18.221748 6059 raft_consensus.cc:2798] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:18.222393 6059 raft_consensus.cc:695] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:18.223145 6059 consensus_queue.cc:237] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.239462 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 01962c5b7ae64f80b998948d18250b4c (table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]) successfully deleted
W20250110 20:57:18.240150 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet c7961f8bd3ff42ffab214a04c704fd26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7961f8bd3ff42ffab214a04c704fd26 already in progress: creating tablet
W20250110 20:57:18.242585 5257 leader_election.cc:343] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.243170 5257 leader_election.cc:304] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.256878 5597 tablet_bootstrap.cc:492] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:18.262270 5594 raft_consensus.cc:357] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.264423 5594 raft_consensus.cc:383] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.267382 5594 raft_consensus.cc:738] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.268980 5597 tablet_bootstrap.cc:654] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.270748 5595 raft_consensus.cc:2238] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:18.271646 5594 consensus_queue.cc:260] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.272910 5595 raft_consensus.cc:2267] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.275739 5595 ts_tablet_manager.cc:1905] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.280581 5594 ts_tablet_manager.cc:1428] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.093s user 0.003s sys 0.002s
I20250110 20:57:18.281383 5594 tablet_bootstrap.cc:492] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:18.281746 6175 raft_consensus.cc:491] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.282246 6175 raft_consensus.cc:513] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.284384 6175 leader_election.cc:290] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:18.289819 6175 raft_consensus.cc:2743] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:18.286751 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:18.295774 5255 leader_election.cc:343] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.298029 5600 tablet_bootstrap.cc:492] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:18.298592 5600 ts_tablet_manager.cc:1397] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.168s user 0.009s sys 0.007s
I20250110 20:57:18.309666 5592 ts_tablet_manager.cc:1918] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:18.310068 5592 log.cc:1198] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/2c1ca1670916401ab0a43685df1b6759
I20250110 20:57:18.310264 5773 ts_tablet_manager.cc:1918] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:18.310779 5592 ts_tablet_manager.cc:1939] T 2c1ca1670916401ab0a43685df1b6759 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:18.310593 5773 log.cc:1198] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/7462d21b91c2441fb9e11383466e9ca7
I20250110 20:57:18.315268 5773 ts_tablet_manager.cc:1939] T 7462d21b91c2441fb9e11383466e9ca7 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:18.315965 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 209) took 2261 ms. Trace:
I20250110 20:57:18.316254 5773 rpcz_store.cc:276] 0110 20:57:16.054666 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.069913 (+ 15247us) service_pool.cc:225] Handling call
0110 20:57:18.111404 (+2041491us) ts_tablet_manager.cc:1145] Deleting tablet 7462d21b91c2441fb9e11383466e9ca7
0110 20:57:18.111438 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.249011 (+137573us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.310050 (+ 61039us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.315916 (+ 5866us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4651,"tablet-delete.queue_time_us":2040801}
I20250110 20:57:18.317471 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 211) took 2008 ms. Trace:
I20250110 20:57:18.317502 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]) successfully deleted
I20250110 20:57:18.317745 5773 rpcz_store.cc:276] 0110 20:57:16.308733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.308844 (+ 111us) service_pool.cc:225] Handling call
0110 20:57:18.317233 (+2008389us) ts_tablet_manager.cc:1145] Deleting tablet e361fb8b47d74548b5a2326d25a9e28c
0110 20:57:18.317271 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.317425 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007742}
I20250110 20:57:18.319442 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 195) took 2674 ms. Trace:
I20250110 20:57:18.323181 5594 tablet_bootstrap.cc:654] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.324280 5592 rpcz_store.cc:276] 0110 20:57:15.644977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.645106 (+ 129us) service_pool.cc:225] Handling call
0110 20:57:18.066451 (+2421345us) ts_tablet_manager.cc:1145] Deleting tablet 2c1ca1670916401ab0a43685df1b6759
0110 20:57:18.066588 (+ 137us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.189796 (+123208us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.261191 (+ 71395us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.319278 (+ 58087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2420547}
I20250110 20:57:18.314330 5135 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:35154 (request call id 54) took 1111 ms. Trace:
W20250110 20:57:18.329427 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet e361fb8b47d74548b5a2326d25a9e28c because the tablet was not found. No further retry: Not found: Tablet not found: e361fb8b47d74548b5a2326d25a9e28c
I20250110 20:57:18.329591 5135 rpcz_store.cc:276] 0110 20:57:17.202621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.357809 (+155188us) service_pool.cc:225] Handling call
0110 20:57:17.488128 (+130319us) spinlock_profiling.cc:243] Waited 29.9 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c9142 00007f8965232cd9 00007f89644c0750 00007f8964586786 00007f8964586960 00007f896453f5e2 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1
0110 20:57:17.962693 (+474565us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b54001324d4. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:18.055506 (+ 92813us) spinlock_profiling.cc:243] Waited 88.6 ms on lock 0x7b54000e0bd4. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:18.182777 (+127271us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400160154. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:18.229688 (+ 46911us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b5400131ad4. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:18.245887 (+ 16199us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b54000d2fd4. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:18.314261 (+ 68374us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1516001152}
I20250110 20:57:18.329283 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 212) took 2020 ms. Trace:
I20250110 20:57:18.334336 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 2c1ca1670916401ab0a43685df1b6759 (table default.Table_792d0af8e9d04ad28d950c7656211372 [id=176f78029038442b932e8c92372948ef]) successfully deleted
I20250110 20:57:18.334298 5773 rpcz_store.cc:276] 0110 20:57:16.309314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.310091 (+ 777us) service_pool.cc:225] Handling call
0110 20:57:18.329041 (+2018950us) ts_tablet_manager.cc:1145] Deleting tablet e27e43461251413b97721323cd935374
0110 20:57:18.329086 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.329239 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2018434}
I20250110 20:57:18.335364 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 196) took 2688 ms. Trace:
W20250110 20:57:18.336155 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet e27e43461251413b97721323cd935374 because the tablet was not found. No further retry: Not found: Tablet not found: e27e43461251413b97721323cd935374
I20250110 20:57:18.337105 5773 tablet_replica.cc:331] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:18.335675 5592 rpcz_store.cc:276] 0110 20:57:15.646769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.646885 (+ 116us) service_pool.cc:225] Handling call
0110 20:57:18.335129 (+2688244us) ts_tablet_manager.cc:1145] Deleting tablet 19d6eca349514b9fb0fff7cd6230027c
0110 20:57:18.335168 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.335310 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2687723}
I20250110 20:57:18.338342 5773 raft_consensus.cc:2238] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
W20250110 20:57:18.338778 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 19d6eca349514b9fb0fff7cd6230027c because the tablet was not found. No further retry: Not found: Tablet not found: 19d6eca349514b9fb0fff7cd6230027c
I20250110 20:57:18.339989 5592 tablet_replica.cc:331] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:18.340024 5773 raft_consensus.cc:2267] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.342672 5592 raft_consensus.cc:2238] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:18.343549 5773 ts_tablet_manager.cc:1905] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.343874 5592 raft_consensus.cc:2267] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.345798 5592 ts_tablet_manager.cc:1905] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.341883 5594 tablet_bootstrap.cc:492] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:18.352279 5597 tablet_bootstrap.cc:492] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:18.352968 5597 ts_tablet_manager.cc:1397] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.006s
I20250110 20:57:18.301081 5600 raft_consensus.cc:357] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.353366 5600 raft_consensus.cc:383] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.353675 5600 raft_consensus.cc:738] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.356038 5597 raft_consensus.cc:357] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.356689 5597 raft_consensus.cc:383] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.361480 5597 raft_consensus.cc:738] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.362145 5597 consensus_queue.cc:260] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.365679 5600 consensus_queue.cc:260] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.367918 5600 ts_tablet_manager.cc:1428] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
I20250110 20:57:18.377208 5597 ts_tablet_manager.cc:1428] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.023s user 0.003s sys 0.001s
I20250110 20:57:18.366802 6145 raft_consensus.cc:491] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.381661 5594 ts_tablet_manager.cc:1397] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.100s user 0.007s sys 0.003s
I20250110 20:57:18.381839 6145 raft_consensus.cc:513] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.384081 6145 leader_election.cc:290] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:18.384476 6145 raft_consensus.cc:491] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.384716 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c44ca47c747544b6afec83b399dcdcb5" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:18.384893 6145 raft_consensus.cc:513] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:18.385847 5399 leader_election.cc:343] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:18.386210 5398 leader_election.cc:343] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: c44ca47c747544b6afec83b399dcdcb5
I20250110 20:57:18.386626 5398 leader_election.cc:304] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.386958 6145 leader_election.cc:290] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:18.387342 6145 raft_consensus.cc:2743] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:18.387279 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ef3773a87e4a23a19ac1e5f603d2ce" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:18.388211 5399 leader_election.cc:343] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.388562 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ef3773a87e4a23a19ac1e5f603d2ce" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:18.389194 5323 raft_consensus.cc:2463] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:18.390221 5398 leader_election.cc:304] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.391101 6205 raft_consensus.cc:2798] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:18.391391 6205 raft_consensus.cc:491] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:18.391662 6205 raft_consensus.cc:3054] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:18.396102 5597 tablet_bootstrap.cc:492] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:18.399950 6205 raft_consensus.cc:513] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.402403 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ef3773a87e4a23a19ac1e5f603d2ce" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:18.402853 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57ef3773a87e4a23a19ac1e5f603d2ce" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:18.403329 5323 raft_consensus.cc:3054] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:18.404395 5399 leader_election.cc:343] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.409482 6205 leader_election.cc:290] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:18.410331 5323 raft_consensus.cc:2463] T 57ef3773a87e4a23a19ac1e5f603d2ce P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:18.411259 5398 leader_election.cc:304] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.412250 6205 raft_consensus.cc:2798] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:18.412689 6205 raft_consensus.cc:695] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:18.413748 6205 consensus_queue.cc:237] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.415211 5773 ts_tablet_manager.cc:1918] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:18.417480 5600 tablet_bootstrap.cc:492] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:18.417819 5773 log.cc:1198] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:18.418540 5773 ts_tablet_manager.cc:1939] T 8d67e94c8f9a414499b5e20a5cc0ead3 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
W20250110 20:57:18.419960 4896 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75862d0008d44a6c8694aa02dae45a80 consensus state has no leader
I20250110 20:57:18.420295 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 213) took 2036 ms. Trace:
I20250110 20:57:18.420789 4896 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_033612db3a3843f999da6a5beb2b8b47" } modify_external_catalogs: true
I20250110 20:57:18.420567 5773 rpcz_store.cc:276] 0110 20:57:16.383880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.383979 (+ 99us) service_pool.cc:225] Handling call
0110 20:57:18.336948 (+1952969us) ts_tablet_manager.cc:1145] Deleting tablet 8d67e94c8f9a414499b5e20a5cc0ead3
0110 20:57:18.336987 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.395909 (+ 58922us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.415005 (+ 19096us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.420240 (+ 5235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938190}
I20250110 20:57:18.421262 4896 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_033612db3a3843f999da6a5beb2b8b47" } modify_external_catalogs: true
I20250110 20:57:18.421666 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 8d67e94c8f9a414499b5e20a5cc0ead3 (table default.Table_653008e13c0c4667a68753a92ae1a596 [id=b13c0c58cff5403295d5cfd2f8935021]) successfully deleted
I20250110 20:57:18.426506 5597 tablet_bootstrap.cc:654] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:18.426594 5257 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 178953us
I20250110 20:57:18.431139 6213 raft_consensus.cc:738] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14d1e2d5c72f498f915c367e8750361f, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.437773 5773 tablet_replica.cc:331] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:18.438681 5773 raft_consensus.cc:2238] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:18.439183 5773 raft_consensus.cc:2267] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.440901 5773 ts_tablet_manager.cc:1905] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.443262 5314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:18.432089 6213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:18.448799 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:18.459628 5594 raft_consensus.cc:357] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.460844 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_1588a911bebe4556aee77ec7ebb160cf [id=2c78ee52e44a46fe877f474a7ecb771b]). Expected version 2 got 0
I20250110 20:57:18.462579 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
W20250110 20:57:18.471000 5111 leader_election.cc:343] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: c6914e258314423c859eb31e85853274
W20250110 20:57:18.471917 5111 leader_election.cc:343] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 527bbddda8184a6eb67c2411ae50db63
I20250110 20:57:18.472649 5111 leader_election.cc:304] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:18.472815 5594 raft_consensus.cc:383] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.473070 5594 raft_consensus.cc:738] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.474148 6253 raft_consensus.cc:2743] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:18.474380 6269 raft_consensus.cc:491] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.474795 6269 raft_consensus.cc:513] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.476601 6269 leader_election.cc:290] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:18.475324 5111 consensus_peers.cc:487] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd0a666892d04e97a4b9967e38ab0946. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:18.477977 5111 consensus_peers.cc:487] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c1ca1670916401ab0a43685df1b6759. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:18.478123 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b954dbccb45b2a6ab7cc5cceadffb" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:18.473776 5594 consensus_queue.cc:260] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.478681 5460 raft_consensus.cc:2463] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:18.479934 5594 ts_tablet_manager.cc:1428] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.098s user 0.004s sys 0.000s
I20250110 20:57:18.480016 5110 leader_election.cc:304] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:18.480715 5594 tablet_bootstrap.cc:492] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:18.480727 6256 raft_consensus.cc:2798] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:18.481016 6256 raft_consensus.cc:491] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:18.481297 6256 raft_consensus.cc:3054] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:18.485158 5600 tablet_bootstrap.cc:654] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.488025 6256 raft_consensus.cc:513] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.491230 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c5b954dbccb45b2a6ab7cc5cceadffb" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:18.491748 5460 raft_consensus.cc:3054] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:18.491811 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f52be800c137445c9aaf68c8654c80b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:18.493142 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f52be800c137445c9aaf68c8654c80b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:18.497989 5460 raft_consensus.cc:2463] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:18.498905 5110 leader_election.cc:304] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:18.499521 6269 raft_consensus.cc:2798] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:18.499639 6256 leader_election.cc:290] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:18.499929 6269 raft_consensus.cc:695] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:18.500597 6269 consensus_queue.cc:237] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.512799 6313 tablet.cc:1721] T cb7f09df8e274f3e9542292d5c1ec6c6 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250110 20:57:18.513556 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5eb75e1886e74a719dc09ccad8e1e284 consensus state has no leader
I20250110 20:57:18.514237 4893 master_service.cc:946] ReplaceTablet: received request to replace tablet 7462d21b91c2441fb9e11383466e9ca7 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:18.525961 5509 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
I20250110 20:57:18.526103 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_2167319943e24d52ad2939094f31e4c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:18.528450 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2167319943e24d52ad2939094f31e4c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:18.529784 6213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:57:18.509462 5595 ts_tablet_manager.cc:1918] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:18.540431 5595 log.cc:1198] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/fde470279464496aa7efdee3800754f3
I20250110 20:57:18.541435 5595 ts_tablet_manager.cc:1939] T fde470279464496aa7efdee3800754f3 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:18.543102 5597 tablet_bootstrap.cc:492] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:18.543432 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 214) took 1006 ms. Trace:
I20250110 20:57:18.543560 5597 ts_tablet_manager.cc:1397] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.148s user 0.009s sys 0.000s
I20250110 20:57:18.543725 5595 rpcz_store.cc:276] 0110 20:57:17.537373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.538382 (+ 1009us) service_pool.cc:225] Handling call
0110 20:57:18.189071 (+650689us) ts_tablet_manager.cc:1145] Deleting tablet fde470279464496aa7efdee3800754f3
0110 20:57:18.189507 (+ 436us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.378290 (+188783us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.465294 (+ 87004us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.543366 (+ 78072us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8969,"tablet-delete.queue_time_us":649830}
I20250110 20:57:18.544967 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet fde470279464496aa7efdee3800754f3 (table default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]) successfully deleted
I20250110 20:57:18.546025 5597 raft_consensus.cc:357] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:18.546412 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 32cf221df30e4e78bd2a8078c26d29ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32cf221df30e4e78bd2a8078c26d29ba already in progress: creating tablet
I20250110 20:57:18.546628 5597 raft_consensus.cc:383] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.546844 5597 raft_consensus.cc:738] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.547338 5597 consensus_queue.cc:260] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.549207 5597 ts_tablet_manager.cc:1428] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250110 20:57:18.556460 5773 ts_tablet_manager.cc:1918] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:18.556941 5592 ts_tablet_manager.cc:1918] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:18.557323 5592 log.cc:1198] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/527bbddda8184a6eb67c2411ae50db63
I20250110 20:57:18.559528 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 4b28befa048244309a06f69b8a3b4b8f
I20250110 20:57:18.561383 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b28befa048244309a06f69b8a3b4b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:18 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:18.561858 5592 ts_tablet_manager.cc:1939] T 527bbddda8184a6eb67c2411ae50db63 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:18.566372 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b28befa048244309a06f69b8a3b4b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:18 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:18.561228 4896 master_service.cc:946] ReplaceTablet: received request to replace tablet d6176c29f01b4ca5888c844c131270f2 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:18.570000 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 198) took 2855 ms. Trace:
I20250110 20:57:18.570375 5592 rpcz_store.cc:276] 0110 20:57:15.714836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.714953 (+ 117us) service_pool.cc:225] Handling call
0110 20:57:18.339541 (+2624588us) ts_tablet_manager.cc:1145] Deleting tablet 527bbddda8184a6eb67c2411ae50db63
0110 20:57:18.339574 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.472021 (+132447us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.556745 (+ 84724us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.569952 (+ 13207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2624047}
I20250110 20:57:18.567561 5148 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b28befa048244309a06f69b8a3b4b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:18 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.569319 6269 consensus_queue.cc:1035] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:57:18.572408 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 4b28befa048244309a06f69b8a3b4b8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b28befa048244309a06f69b8a3b4b8f already in progress: creating tablet
W20250110 20:57:18.510437 5257 leader_election.cc:343] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.574124 5257 leader_election.cc:304] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.574858 6175 raft_consensus.cc:2743] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:18.575718 5257 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 65304.7us
I20250110 20:57:18.577751 5773 log.cc:1198] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/c951d5ca47e8409db646b43bb04a7034
I20250110 20:57:18.581979 5592 tablet_replica.cc:331] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:18.582084 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 527bbddda8184a6eb67c2411ae50db63 (table default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]) successfully deleted
I20250110 20:57:18.583195 5170 raft_consensus.cc:3054] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:18.583913 5773 ts_tablet_manager.cc:1939] T c951d5ca47e8409db646b43bb04a7034 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:18.584602 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 214) took 2190 ms. Trace:
I20250110 20:57:18.573683 6205 raft_consensus.cc:491] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.587455 6205 raft_consensus.cc:513] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.590291 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:18.584884 5773 rpcz_store.cc:276] 0110 20:57:16.394549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.394659 (+ 110us) service_pool.cc:225] Handling call
0110 20:57:18.437630 (+2042971us) ts_tablet_manager.cc:1145] Deleting tablet c951d5ca47e8409db646b43bb04a7034
0110 20:57:18.437671 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.515751 (+ 78080us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.556266 (+ 40515us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.584554 (+ 28288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2038713}
W20250110 20:57:18.591562 5399 leader_election.cc:343] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.592033 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:18.592659 5323 raft_consensus.cc:2463] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:18.593602 5398 leader_election.cc:304] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.594318 6251 raft_consensus.cc:2798] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:18.594616 6251 raft_consensus.cc:491] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:18.594890 6251 raft_consensus.cc:3054] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:18.598585 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 215) took 2181 ms. Trace:
I20250110 20:57:18.521535 6311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:18.601143 6311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:18.603634 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet c951d5ca47e8409db646b43bb04a7034 (table default.Table_2a36a6d3c1c843eba552b6fd97984099 [id=723ca40b46694de9802850cb89dca66f]) successfully deleted
I20250110 20:57:18.604642 5773 rpcz_store.cc:276] 0110 20:57:16.417330 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.417457 (+ 127us) service_pool.cc:225] Handling call
0110 20:57:18.598259 (+2180802us) ts_tablet_manager.cc:1145] Deleting tablet c44ca47c747544b6afec83b399dcdcb5
0110 20:57:18.598297 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.598536 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2180280}
I20250110 20:57:18.599560 5597 tablet_bootstrap.cc:492] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:18.605645 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet c44ca47c747544b6afec83b399dcdcb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c44ca47c747544b6afec83b399dcdcb5 already in progress: creating tablet
I20250110 20:57:18.606500 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_b3b9ceb781ee4ac0ac6c046d253749c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:18.607389 6251 raft_consensus.cc:513] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:18.607887 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3b9ceb781ee4ac0ac6c046d253749c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:18.610067 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:18.614197 5110 consensus_peers.cc:487] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:18.615412 5323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:18.615926 5323 raft_consensus.cc:3054] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:18.615957 5399 leader_election.cc:343] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:18.617456 5399 consensus_peers.cc:487] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:18.617825 6251 leader_election.cc:290] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:18.626194 6312 tablet.cc:1721] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:18.626443 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2a36a6d3c1c843eba552b6fd97984099" } new_table_name: "default.Table_6ca3c213749745caa228b62726499ecd" modify_external_catalogs: true
I20250110 20:57:18.629119 5592 raft_consensus.cc:2238] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:18.630517 6205 leader_election.cc:290] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:18.630977 5592 pending_rounds.cc:62] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:18.631217 5592 pending_rounds.cc:69] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878642619277312 op_type: NO_OP noop_request { }
I20250110 20:57:18.631526 5592 raft_consensus.cc:2883] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:18.631820 5592 raft_consensus.cc:2267] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.635236 4896 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:18.635519 5323 raft_consensus.cc:2463] T eee23ab7ecce465ca1153d22c40bf935 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:18.636430 4896 catalog_manager.cc:6596] ReplaceTablet: tablet d6176c29f01b4ca5888c844c131270f2 deleted and replaced by tablet 700374c672fe43c9a45f34d0255b8683
I20250110 20:57:18.636899 5438 tablet_service.cc:1514] Processing DeleteTablet for tablet d6176c29f01b4ca5888c844c131270f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 700374c672fe43c9a45f34d0255b8683) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:18.637466 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet d6176c29f01b4ca5888c844c131270f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 700374c672fe43c9a45f34d0255b8683) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:18.636801 5398 leader_election.cc:304] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.638095 5595 tablet_replica.cc:331] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:18.638968 5595 raft_consensus.cc:2238] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:18.632097 6121 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:18.639405 5148 tablet_service.cc:1514] Processing DeleteTablet for tablet d6176c29f01b4ca5888c844c131270f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 700374c672fe43c9a45f34d0255b8683) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:18.639710 6175 raft_consensus.cc:491] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.639906 5595 raft_consensus.cc:2267] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.640125 6175 raft_consensus.cc:513] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.641917 6175 leader_election.cc:290] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:18.642359 6205 raft_consensus.cc:2798] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:18.642933 6205 raft_consensus.cc:695] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:18.647704 6205 consensus_queue.cc:237] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.651074 5595 ts_tablet_manager.cc:1905] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.655511 5600 tablet_bootstrap.cc:492] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:18.655464 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484a3aebaafb4f2aa47f009b773209fe" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:18.655923 5600 ts_tablet_manager.cc:1397] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.239s user 0.010s sys 0.000s
W20250110 20:57:18.656762 5255 leader_election.cc:343] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.658162 5600 raft_consensus.cc:357] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.660391 5592 ts_tablet_manager.cc:1905] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.668624 5597 tablet_bootstrap.cc:654] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.673180 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e50ee362eaec42db8d62723516fded43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:18.673906 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 216) took 2155 ms. Trace:
I20250110 20:57:18.674219 5773 rpcz_store.cc:276] 0110 20:57:16.518050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.527094 (+ 9044us) service_pool.cc:225] Handling call
0110 20:57:18.667290 (+2140196us) ts_tablet_manager.cc:1145] Deleting tablet e508061e93cc430cb88f6d10d9684f00
0110 20:57:18.673456 (+ 6166us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.673854 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2138957}
W20250110 20:57:18.674544 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e50ee362eaec42db8d62723516fded43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:18.674959 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484a3aebaafb4f2aa47f009b773209fe" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:18.692195 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet e508061e93cc430cb88f6d10d9684f00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e508061e93cc430cb88f6d10d9684f00 already in progress: creating tablet
I20250110 20:57:18.693233 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 218) took 1809 ms. Trace:
I20250110 20:57:18.693990 5773 rpcz_store.cc:276] 0110 20:57:16.883892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.884133 (+ 241us) service_pool.cc:225] Handling call
0110 20:57:18.692918 (+1808785us) ts_tablet_manager.cc:1145] Deleting tablet a95f05fc5afe4877a606b501dbd09551
0110 20:57:18.692957 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.693180 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808236}
I20250110 20:57:18.696087 5773 tablet_replica.cc:331] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:18.697284 5773 raft_consensus.cc:2238] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 20:57:18.695128 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet a95f05fc5afe4877a606b501dbd09551 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a95f05fc5afe4877a606b501dbd09551 already in progress: creating tablet
I20250110 20:57:18.731621 4896 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_156534e2fa0e4fa7bd3eb498d05a4626" } new_table_name: "default.Table_6620f092e4b4461a8038cb09f49b2f81" modify_external_catalogs: true
I20250110 20:57:18.746534 5594 tablet_bootstrap.cc:654] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.713999 6175 raft_consensus.cc:491] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.743427 5595 ts_tablet_manager.cc:1918] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:18.755095 5595 log.cc:1198] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:18.755019 6175 raft_consensus.cc:513] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.755909 5595 ts_tablet_manager.cc:1939] T d6176c29f01b4ca5888c844c131270f2 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:18.750252 6331 tablet.cc:1721] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:18.758021 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]) successfully deleted
W20250110 20:57:18.767599 4896 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a68d4b44975b436fb1b4a9ee6ff290d2 consensus state has no leader
I20250110 20:57:18.769035 5600 raft_consensus.cc:383] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.777627 5600 raft_consensus.cc:738] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:18.754431 5257 leader_election.cc:343] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.756893 6175 leader_election.cc:290] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:18.782114 5257 leader_election.cc:304] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.787014 6175 raft_consensus.cc:2743] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:18.768337 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae52ea080f54d228b6b76922248ed5a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:18.790254 5255 leader_election.cc:343] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.791154 5600 consensus_queue.cc:260] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:18.706213 5111 leader_election.cc:343] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 5c5b954dbccb45b2a6ab7cc5cceadffb
W20250110 20:57:18.797868 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 91674.3us
W20250110 20:57:18.799201 5111 leader_election.cc:343] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 5c5b954dbccb45b2a6ab7cc5cceadffb
I20250110 20:57:18.802920 5773 raft_consensus.cc:2267] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:18.788683 5597 tablet_bootstrap.cc:492] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:18.813345 6338 tablet.cc:1721] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:18.813954 5597 ts_tablet_manager.cc:1397] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.215s user 0.004s sys 0.005s
I20250110 20:57:18.784480 6332 tablet.cc:1721] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:18.826953 6175 raft_consensus.cc:491] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.827400 6175 raft_consensus.cc:513] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.829241 6175 leader_election.cc:290] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:18.831174 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:18.832342 5255 leader_election.cc:343] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.839248 6337 tablet.cc:1721] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:18.818583 5600 ts_tablet_manager.cc:1428] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.162s user 0.005s sys 0.000s
I20250110 20:57:18.843735 5600 tablet_bootstrap.cc:492] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:18.844555 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae52ea080f54d228b6b76922248ed5a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:18.848841 5600 tablet_bootstrap.cc:654] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.850867 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:18.852336 5594 tablet_bootstrap.cc:492] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
W20250110 20:57:18.853451 5257 leader_election.cc:343] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.853979 5257 leader_election.cc:304] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.858001 6306 raft_consensus.cc:2743] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:18.870035 5773 ts_tablet_manager.cc:1905] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:18.878846 5597 raft_consensus.cc:357] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.879601 5597 raft_consensus.cc:383] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:18.879887 5597 raft_consensus.cc:738] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:18.880537 5597 consensus_queue.cc:260] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.884213 5509 heartbeater.cc:502] Master 127.4.171.252:37233 was elected leader, sending a full tablet report...
I20250110 20:57:18.894654 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:18.895380 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (table default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145]). Expected version 1 got 0
I20250110 20:57:18.896261 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:18.899896 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 78375cb83ab04471af009dbf231fc28d (DEFAULT_TABLE table=default.Table_8fefcbeeff174088ad6b60ddc002689d [id=91b808a5b0024c889cbb87ac3ff8a6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.900748 5438 tablet_service.cc:1467] Processing CreateTablet for tablet 78375cb83ab04471af009dbf231fc28d (DEFAULT_TABLE table=default.Table_8fefcbeeff174088ad6b60ddc002689d [id=91b808a5b0024c889cbb87ac3ff8a6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.901713 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78375cb83ab04471af009dbf231fc28d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:18.906668 5257 leader_election.cc:343] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:18.907135 5257 leader_election.cc:304] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:18.893261 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 78375cb83ab04471af009dbf231fc28d (DEFAULT_TABLE table=default.Table_8fefcbeeff174088ad6b60ddc002689d [id=91b808a5b0024c889cbb87ac3ff8a6d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.907773 6175 raft_consensus.cc:2743] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:18.893484 6059 raft_consensus.cc:491] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46005a5920cc4a3697a5c61b1c271384)
I20250110 20:57:18.908548 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78375cb83ab04471af009dbf231fc28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.908411 6059 raft_consensus.cc:513] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.910986 6059 leader_election.cc:290] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:18.913020 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 51fae5f59d4642a09acee5a687b91717 (DEFAULT_TABLE table=default.Table_53bbb78e4859491ba4013f5dd868d449 [id=e4e3948de85c4db480e29af5b2d6c4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.913969 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51fae5f59d4642a09acee5a687b91717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.916927 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 51fae5f59d4642a09acee5a687b91717 (DEFAULT_TABLE table=default.Table_53bbb78e4859491ba4013f5dd868d449 [id=e4e3948de85c4db480e29af5b2d6c4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.917388 5144 tablet_service.cc:1467] Processing CreateTablet for tablet c38e54701a2844b1977ec1a3a40a96dd (DEFAULT_TABLE table=default.Table_d390bf037cd241728651d40178602a0d [id=29ae4ee3abb848f986968e5688aebe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.917821 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51fae5f59d4642a09acee5a687b91717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.918239 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38e54701a2844b1977ec1a3a40a96dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.918615 5292 tablet_service.cc:1467] Processing CreateTablet for tablet 51fae5f59d4642a09acee5a687b91717 (DEFAULT_TABLE table=default.Table_53bbb78e4859491ba4013f5dd868d449 [id=e4e3948de85c4db480e29af5b2d6c4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.926908 5597 ts_tablet_manager.cc:1428] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.107s user 0.004s sys 0.000s
I20250110 20:57:18.927681 5597 tablet_bootstrap.cc:492] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:18.930028 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:18.932674 4894 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet cb7f09df8e274f3e9542292d5c1ec6c6 (table default.Table_1588a911bebe4556aee77ec7ebb160cf [id=2c78ee52e44a46fe877f474a7ecb771b]). Expected version 2 got 0
I20250110 20:57:18.932732 5294 tablet_service.cc:1467] Processing CreateTablet for tablet c38e54701a2844b1977ec1a3a40a96dd (DEFAULT_TABLE table=default.Table_d390bf037cd241728651d40178602a0d [id=29ae4ee3abb848f986968e5688aebe9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.933908 5442 tablet_service.cc:1467] Processing CreateTablet for tablet c38e54701a2844b1977ec1a3a40a96dd (DEFAULT_TABLE table=default.Table_d390bf037cd241728651d40178602a0d [id=29ae4ee3abb848f986968e5688aebe9f]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:18.934659 5399 consensus_peers.cc:487] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c951d5ca47e8409db646b43bb04a7034. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:18.934806 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38e54701a2844b1977ec1a3a40a96dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.925319 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:18.946627 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (table default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145]). Expected version 1 got 0
I20250110 20:57:18.947587 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_5675215f702a41ab934b6baf97fd0431 [id=1c2b9e8ff5d341c380b50023c577d0aa]). Expected version 2 got 0
I20250110 20:57:18.948137 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:18.949580 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:18.852756 5594 ts_tablet_manager.cc:1397] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.372s user 0.009s sys 0.001s
I20250110 20:57:18.952832 5594 raft_consensus.cc:357] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:18.955276 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:18.955801 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff]). Expected version 1 got 0
I20250110 20:57:18.956835 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:18.959228 5148 tablet_service.cc:1467] Processing CreateTablet for tablet 937831ec98d74007b8207ecb30843550 (DEFAULT_TABLE table=default.Table_4a1acf754c984e86ac40fd7ab1509b2b [id=33ffc109fdf34bc1b9de49832d42bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.962550 5597 tablet_bootstrap.cc:654] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.965487 6175 raft_consensus.cc:491] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:18.965903 6175 raft_consensus.cc:513] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:18.968479 6175 leader_election.cc:290] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:18.970542 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 937831ec98d74007b8207ecb30843550 (DEFAULT_TABLE table=default.Table_4a1acf754c984e86ac40fd7ab1509b2b [id=33ffc109fdf34bc1b9de49832d42bee5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:18.981707 5257 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 75063.6us
I20250110 20:57:18.981861 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab3f2a8655c4f299f3fc62d11fad439 (DEFAULT_TABLE table=default.Table_bdd0ff0e681b4f57ba64a67252186056 [id=65c1840b0b34455f8d0c67e58b795263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.983548 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d1d5fba23d8411aa7cbf4781f1c7fc1" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:18.985620 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab3f2a8655c4f299f3fc62d11fad439 (DEFAULT_TABLE table=default.Table_bdd0ff0e681b4f57ba64a67252186056 [id=65c1840b0b34455f8d0c67e58b795263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.984563 4896 catalog_manager.cc:5526] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:18.986258 5773 ts_tablet_manager.cc:1918] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:18.986613 5773 log.cc:1198] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/01962c5b7ae64f80b998948d18250b4c
I20250110 20:57:18.989182 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 5bde9f66fa4647558d50e1b2febd4335 (DEFAULT_TABLE table=default.Table_a262069587954407a948374ed4c1b18f [id=125620153cf443888a18ce75d09c8649]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.989850 5773 ts_tablet_manager.cc:1939] T 01962c5b7ae64f80b998948d18250b4c P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:18.989773 4896 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (table default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145]). Expected version 1 got 0
I20250110 20:57:18.990353 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78375cb83ab04471af009dbf231fc28d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.990449 4896 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]). Expected version 1 got 0
I20250110 20:57:18.992173 4896 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet f2cbb46ea11c4e7d8bc5993ccdc52247 (table default.Table_ae86414449b14c9aba13ee321887871a [id=9c593f8a26874512a72f0bfc20975c86]). Expected version 1 got 0
I20250110 20:57:18.987715 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 5bde9f66fa4647558d50e1b2febd4335 (DEFAULT_TABLE table=default.Table_a262069587954407a948374ed4c1b18f [id=125620153cf443888a18ce75d09c8649]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.986658 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57941aee12a4ebea2b47f4727fc6eee" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:18.993041 4896 catalog_manager.cc:5526] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:18.994168 5292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51fae5f59d4642a09acee5a687b91717. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.995154 5152 tablet_service.cc:1467] Processing CreateTablet for tablet 97dfd451501d49909b3d163a53548ce4 (DEFAULT_TABLE table=default.Table_99db920d921840a19db98cc103c4fdab [id=1c212ce6a4d8478987cffb863cafb48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.995635 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c38e54701a2844b1977ec1a3a40a96dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.997040 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 937831ec98d74007b8207ecb30843550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.998518 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab3f2a8655c4f299f3fc62d11fad439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.998903 4896 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:18.999639 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bde9f66fa4647558d50e1b2febd4335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.005038 5147 tablet_service.cc:1467] Processing CreateTablet for tablet 36bca826ec144f77b96b4a41277fd6d5 (DEFAULT_TABLE table=default.Table_c7eb26eb1ef541e7853e3445290e6d49 [id=d8003ca561f74cbda31f38b6e9c51f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:18.999491 4896 catalog_manager.cc:5526] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:19.009958 4896 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:19.010797 4892 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 41de354807004e26bfb147f1d420037e
I20250110 20:57:19.012081 4892 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 266) took 1493 ms. Trace:
I20250110 20:57:19.012683 5146 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de354807004e26bfb147f1d420037e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.015074 5146 tablet_service.cc:1467] Processing CreateTablet for tablet d2b789aad72643149da01b9a607e2ce5 (DEFAULT_TABLE table=default.Table_754923f444f045e9b7d6c41c1b11bef0 [id=fa79e948d9db4cc89c8845b76c1c4e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.025322 5154 tablet_service.cc:1467] Processing CreateTablet for tablet bf4fbb2af0b84d35bd279ea2d5dd4ede (DEFAULT_TABLE table=default.Table_68433248a92d4f90bf5f613e7722a04c [id=edc9d679d92b49129d6ad5e695d96b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.030227 5299 tablet_service.cc:1467] Processing CreateTablet for tablet 97dfd451501d49909b3d163a53548ce4 (DEFAULT_TABLE table=default.Table_99db920d921840a19db98cc103c4fdab [id=1c212ce6a4d8478987cffb863cafb48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.030512 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 36bca826ec144f77b96b4a41277fd6d5 (DEFAULT_TABLE table=default.Table_c7eb26eb1ef541e7853e3445290e6d49 [id=d8003ca561f74cbda31f38b6e9c51f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.030683 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet fde470279464496aa7efdee3800754f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.031299 5299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97dfd451501d49909b3d163a53548ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.031697 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de354807004e26bfb147f1d420037e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.031948 5155 tablet_service.cc:1467] Processing CreateTablet for tablet ac9a20eb4dcb40d7b2e1dd1e5ed2edbb (DEFAULT_TABLE table=default.Table_43a94d9dc0074febbfe6afa4d4c9716d [id=0387c31491b6494580f9b02da7b0b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.032264 5595 tablet_replica.cc:331] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:19.032420 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bca826ec144f77b96b4a41277fd6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.032992 5595 raft_consensus.cc:2238] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:19.033490 5595 raft_consensus.cc:2267] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:19.035923 5595 ts_tablet_manager.cc:1905] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:19.038038 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 017a93be79ad49f0a72857cae4ada768 (DEFAULT_TABLE table=default.Table_36d9c90774f84119abe713d7b6331a49 [id=1441f7755521427390ee18c201f2421b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.038901 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet 637a39e4264b40dd9480dc13e37359d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.039245 5150 tablet_service.cc:1467] Processing CreateTablet for tablet b5becf70b58847559b70ce27edc30398 (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.039855 5298 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e43461251413b97721323cd935374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.045142 5149 tablet_service.cc:1467] Processing CreateTablet for tablet 61d67ac398624f05b5a51e61d25e3e35 (DEFAULT_TABLE table=default.Table_f21a12015eab4635890b871de037c960 [id=3498f91140e34fb9b5ff861c40522ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.012362 4892 rpcz_store.cc:276] 0110 20:57:17.518219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.539924 (+ 21705us) service_pool.cc:225] Handling call
0110 20:57:17.541008 (+ 1084us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:17.602966 (+ 61958us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:17.603092 (+ 126us) catalog_manager.cc:2762] Locking tablets
0110 20:57:17.617894 (+ 14802us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:17.918695 (+300801us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:17.918894 (+ 199us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:19.010711 (+1091817us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:19.012021 (+ 1310us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:17.887295 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:17.887627 (+ 332us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:17.887647 (+ 20us) write_op.cc:435] Acquired schema lock
0110 20:57:17.887658 (+ 11us) tablet.cc:659] Decoding operations
0110 20:57:17.887949 (+ 291us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:17.887982 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:17.888003 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:17.888233 (+ 230us) tablet.cc:698] Row locks acquired
0110 20:57:17.888245 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:17.888350 (+ 105us) write_op.cc:270] Start()
0110 20:57:17.888417 (+ 67us) write_op.cc:276] Timestamp: P: 1736542637888290 usec, L: 0
0110 20:57:17.888431 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:17.889000 (+ 569us) log.cc:844] Serialized 796 byte log entry
0110 20:57:17.911911 (+ 22911us) op_driver.cc:464] REPLICATION: finished
0110 20:57:17.912734 (+ 823us) write_op.cc:301] APPLY: starting
0110 20:57:17.912785 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:17.912880 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:17.912969 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:17.913216 (+ 247us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:17.913342 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:17.913385 (+ 43us) write_op.cc:312] APPLY: finished
0110 20:57:17.913558 (+ 173us) log.cc:844] Serialized 29 byte log entry
0110 20:57:17.913686 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.913747 (+ 61us) write_op.cc:454] Released schema lock
0110 20:57:17.913775 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:17.913974 (+ 199us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.914001 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":690,"apply.run_cpu_time_us":1207,"apply.run_wall_time_us":1207,"num_ops":2,"prepare.queue_time_us":265288,"prepare.run_cpu_time_us":1793,"prepare.run_wall_time_us":1888,"replication_time_us":23378}]]}
I20250110 20:57:19.053072 4892 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_31eb7686cc4b4549bab76f9b01c9e311" } modify_external_catalogs: true
I20250110 20:57:19.054806 4892 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_31eb7686cc4b4549bab76f9b01c9e311" } modify_external_catalogs: true
I20250110 20:57:19.055632 5143 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9672dc9cbb4e27bf6b01f513b6ccb8 (DEFAULT_TABLE table=default.Table_82e8a56b6f35429a9996b1622048bf8f [id=6e78cec050ce42a995a1cbb58325a4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.065973 5142 tablet_service.cc:1467] Processing CreateTablet for tablet d26c2bd25161489baa156e78128e35aa (DEFAULT_TABLE table=default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.066097 5298 tablet_service.cc:1467] Processing CreateTablet for tablet d2b789aad72643149da01b9a607e2ce5 (DEFAULT_TABLE table=default.Table_754923f444f045e9b7d6c41c1b11bef0 [id=fa79e948d9db4cc89c8845b76c1c4e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.067102 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b789aad72643149da01b9a607e2ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.070009 5141 tablet_service.cc:1467] Processing CreateTablet for tablet e3e8cca02bf54c5086ecc8562c354d18 (DEFAULT_TABLE table=default.Table_b26866e23984441f96358c0d662acea3 [id=37073a31306f45e9b0e073cecbf1d05d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:19.072023 4898 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 60) took 2704 ms (client timeout 1000 ms). Trace:
I20250110 20:57:19.078627 5302 tablet_service.cc:1467] Processing CreateTablet for tablet ac9a20eb4dcb40d7b2e1dd1e5ed2edbb (DEFAULT_TABLE table=default.Table_43a94d9dc0074febbfe6afa4d4c9716d [id=0387c31491b6494580f9b02da7b0b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.079576 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9a20eb4dcb40d7b2e1dd1e5ed2edbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.079356 5300 tablet_service.cc:1467] Processing CreateTablet for tablet b5becf70b58847559b70ce27edc30398 (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.080348 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5becf70b58847559b70ce27edc30398. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.081046 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 017a93be79ad49f0a72857cae4ada768 (DEFAULT_TABLE table=default.Table_36d9c90774f84119abe713d7b6331a49 [id=1441f7755521427390ee18c201f2421b]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:19.081604 4891 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 61) took 1607 ms (client timeout 1000 ms). Trace:
I20250110 20:57:19.082024 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 017a93be79ad49f0a72857cae4ada768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.083243 5140 tablet_service.cc:1467] Processing CreateTablet for tablet 873b9a1c522d4f7895740db18a537a1b (DEFAULT_TABLE table=default.Table_504da40a6cc54bb4b377ba25e145e4e4 [id=c450ef62b2b44988b1162f0bb99b260f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.090528 5139 tablet_service.cc:1467] Processing CreateTablet for tablet 55542f8735d141789205e602651d38c6 (DEFAULT_TABLE table=default.Table_1e56771d3584418e8e38ebbff845ad4b [id=a3ecf6fce8db49cca87185958657ad9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.097705 5297 tablet_service.cc:1467] Processing CreateTablet for tablet bf4fbb2af0b84d35bd279ea2d5dd4ede (DEFAULT_TABLE table=default.Table_68433248a92d4f90bf5f613e7722a04c [id=edc9d679d92b49129d6ad5e695d96b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.098959 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0cee6de1f94451ba3530f684226055 (DEFAULT_TABLE table=default.Table_e7d42f471e7a4c3999d9b4ed548ae2f6 [id=79d7e17217e74200867a6656410916b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.101778 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4fbb2af0b84d35bd279ea2d5dd4ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.103794 4892 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 32a5af6dfe2447b0b2ad18cad224412c
I20250110 20:57:19.104665 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_78b6d1b2ffe54b66aa5e067af6013c12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.106518 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78b6d1b2ffe54b66aa5e067af6013c12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.110795 6175 raft_consensus.cc:491] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:19.111300 6175 raft_consensus.cc:513] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.113524 6175 leader_election.cc:290] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:19.118238 5137 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9c229f569c4468bf43e7e880f7f0df (DEFAULT_TABLE table=default.Table_e3046e912cd243e591470e603e86c4d5 [id=81cce18d8d0e4c5fbb6dc13bc244cfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.134730 5295 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9672dc9cbb4e27bf6b01f513b6ccb8 (DEFAULT_TABLE table=default.Table_82e8a56b6f35429a9996b1622048bf8f [id=6e78cec050ce42a995a1cbb58325a4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.135500 5291 tablet_service.cc:1514] Processing DeleteTablet for tablet c44ca47c747544b6afec83b399dcdcb5 with delete_type TABLET_DATA_DELETED (replaced by tablet d26c2bd25161489baa156e78128e35aa) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.135803 5295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9672dc9cbb4e27bf6b01f513b6ccb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.136600 5291 tablet_service.cc:1514] Processing DeleteTablet for tablet 7462d21b91c2441fb9e11383466e9ca7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7f8713c1e5a743a489de11bc65857bff) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.137591 5302 tablet_service.cc:1514] Processing DeleteTablet for tablet f3104928188c489d93bfd499d148fa5d with delete_type TABLET_DATA_DELETED (replaced by tablet 29bbd640cb8e4adba77079181e8d1871) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.138016 5291 tablet_service.cc:1467] Processing CreateTablet for tablet e3e8cca02bf54c5086ecc8562c354d18 (DEFAULT_TABLE table=default.Table_b26866e23984441f96358c0d662acea3 [id=37073a31306f45e9b0e073cecbf1d05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.139084 5291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e8cca02bf54c5086ecc8562c354d18. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.115983 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95b52e115a8e4bae95da0ef9d5afff8c consensus state has no leader
I20250110 20:57:19.143723 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e7f52abe4b384cb2abf0e1b9ee4abdc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.145040 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7f52abe4b384cb2abf0e1b9ee4abdc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.148195 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 61d67ac398624f05b5a51e61d25e3e35 (DEFAULT_TABLE table=default.Table_f21a12015eab4635890b871de037c960 [id=3498f91140e34fb9b5ff861c40522ba3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:19.132367 4899 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 52) took 2662 ms (client timeout 1000 ms). Trace:
W20250110 20:57:19.153026 4886 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 95b52e115a8e4bae95da0ef9d5afff8c (table default.Table_6ca3c213749745caa228b62726499ecd [id=723ca40b46694de9802850cb89dca66f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 95b52e115a8e4bae95da0ef9d5afff8c consensus state has no leader
W20250110 20:57:19.133865 4897 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 43) took 924 ms (client timeout 1000 ms). Trace:
W20250110 20:57:19.153553 4897 rpcz_store.cc:269] 0110 20:57:18.209375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.220574 (+ 11199us) service_pool.cc:225] Handling call
0110 20:57:18.234262 (+ 13688us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:19.133811 (+899549us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":16,"spinlock_wait_cycles":4413184}
I20250110 20:57:19.156257 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d67ac398624f05b5a51e61d25e3e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.166597 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d28fb676474445e5bc7377940f7a4c3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:19.178093 5302 tablet_service.cc:1467] Processing CreateTablet for tablet 55542f8735d141789205e602651d38c6 (DEFAULT_TABLE table=default.Table_1e56771d3584418e8e38ebbff845ad4b [id=a3ecf6fce8db49cca87185958657ad9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.178668 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0cee6de1f94451ba3530f684226055 (DEFAULT_TABLE table=default.Table_e7d42f471e7a4c3999d9b4ed548ae2f6 [id=79d7e17217e74200867a6656410916b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.179083 5302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55542f8735d141789205e602651d38c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.179268 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d28fb676474445e5bc7377940f7a4c3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.179680 5290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0cee6de1f94451ba3530f684226055. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.081902 4891 rpcz_store.cc:269] 0110 20:57:17.474119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.529294 (+ 55175us) service_pool.cc:225] Handling call
0110 20:57:17.593374 (+ 64080us) spinlock_profiling.cc:243] Waited 29.7 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:19.081543 (+1488169us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:18.344073 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:18.344546 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:18.344569 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:18.344581 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:18.344803 (+ 222us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:18.344834 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:18.344855 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:18.344980 (+ 125us) tablet.cc:698] Row locks acquired
0110 20:57:18.344992 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:18.345087 (+ 95us) write_op.cc:270] Start()
0110 20:57:18.345161 (+ 74us) write_op.cc:276] Timestamp: P: 1736542638345032 usec, L: 0
0110 20:57:18.345174 (+ 13us) op_driver.cc:348] REPLICATION: starting
0110 20:57:18.345635 (+ 461us) log.cc:844] Serialized 533 byte log entry
0110 20:57:18.417928 (+ 72293us) op_driver.cc:464] REPLICATION: finished
0110 20:57:18.418408 (+ 480us) write_op.cc:301] APPLY: starting
0110 20:57:18.418453 (+ 45us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:18.418525 (+ 72us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:18.418609 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:18.418726 (+ 117us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:18.418832 (+ 106us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:18.418868 (+ 36us) write_op.cc:312] APPLY: finished
0110 20:57:18.418989 (+ 121us) log.cc:844] Serialized 23 byte log entry
0110 20:57:18.419090 (+ 101us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:18.419137 (+ 47us) write_op.cc:454] Released schema lock
0110 20:57:18.419161 (+ 24us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:18.427834 (+ 8673us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:18.427858 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":22968,"reported_tablets":21,"spinlock_wait_cycles":68315520,"child_traces":[["op",{"apply.queue_time_us":364,"apply.run_cpu_time_us":896,"apply.run_wall_time_us":897,"num_ops":1,"prepare.queue_time_us":261296,"prepare.run_cpu_time_us":1793,"prepare.run_wall_time_us":83752,"raft.queue_time_us":1056,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":203,"replication_time_us":72680}]]}
I20250110 20:57:19.186798 6312 tablet.cc:1721] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:19.139328 4895 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_1588a911bebe4556aee77ec7ebb160cf [id=2c78ee52e44a46fe877f474a7ecb771b] alter complete (version 2)
W20250110 20:57:19.129287 4894 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 53) took 1427 ms (client timeout 1000 ms). Trace:
W20250110 20:57:19.190320 4894 rpcz_store.cc:269] 0110 20:57:17.701439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.701558 (+ 119us) service_pool.cc:225] Handling call
0110 20:57:17.753195 (+ 51637us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:19.129224 (+1376029us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":25,"spinlock_wait_cycles":107879296}
I20250110 20:57:19.191520 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f88b13f73f784b8890702e20bc102be2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:19.193727 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 937831ec98d74007b8207ecb30843550. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.194254 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f88b13f73f784b8890702e20bc102be2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.169667 4893 master_service.cc:946] ReplaceTablet: received request to replace tablet fde470279464496aa7efdee3800754f3 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:19.197996 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.198525 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.198926 5597 tablet_bootstrap.cc:492] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:19.198850 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_0b1d1f4267df44e6a2555f1080545d59" } new_table_name: "default.Table_2d69d3afb9ca476cb5b329145ef8f74c" modify_external_catalogs: true
I20250110 20:57:19.198982 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet c951d5ca47e8409db646b43bb04a7034 with delete_type TABLET_DATA_DELETED (replaced by tablet 95b52e115a8e4bae95da0ef9d5afff8c) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.199298 5597 ts_tablet_manager.cc:1397] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.272s user 0.032s sys 0.008s
I20250110 20:57:19.199483 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.199946 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet c6914e258314423c859eb31e85853274 with delete_type TABLET_DATA_DELETED (replaced by tablet a687769f69d844cc8564e61dc324d2b2) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.200398 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1ca1670916401ab0a43685df1b6759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.200840 5144 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:19.172900 4900 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 51) took 3719 ms (client timeout 1000 ms). Trace:
W20250110 20:57:19.072777 4898 rpcz_store.cc:269] 0110 20:57:16.367350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.367447 (+ 97us) service_pool.cc:225] Handling call
0110 20:57:16.437819 (+ 70372us) spinlock_profiling.cc:243] Waited 63.2 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:19.071960 (+2634141us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:17.308932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:17.309274 (+ 342us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:17.309298 (+ 24us) write_op.cc:435] Acquired schema lock
0110 20:57:17.309309 (+ 11us) tablet.cc:659] Decoding operations
0110 20:57:17.309608 (+ 299us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:17.309645 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:17.309666 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:17.309896 (+ 230us) tablet.cc:698] Row locks acquired
0110 20:57:17.309909 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:17.310016 (+ 107us) write_op.cc:270] Start()
0110 20:57:17.310092 (+ 76us) write_op.cc:276] Timestamp: P: 1736542637309951 usec, L: 0
0110 20:57:17.310107 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:17.310531 (+ 424us) log.cc:844] Serialized 913 byte log entry
0110 20:57:17.318260 (+ 7729us) op_driver.cc:464] REPLICATION: finished
0110 20:57:17.318562 (+ 302us) write_op.cc:301] APPLY: starting
0110 20:57:17.318609 (+ 47us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:17.318705 (+ 96us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:17.318790 (+ 85us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:17.319578 (+ 788us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:17.319704 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:17.319741 (+ 37us) write_op.cc:312] APPLY: finished
0110 20:57:17.319920 (+ 179us) log.cc:844] Serialized 29 byte log entry
0110 20:57:17.320048 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.320108 (+ 60us) write_op.cc:454] Released schema lock
0110 20:57:17.320133 (+ 25us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:17.339252 (+ 19119us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.339277 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":24515,"reported_tablets":22,"spinlock_wait_cycles":145309568,"child_traces":[["op",{"apply.queue_time_us":182,"apply.run_cpu_time_us":1731,"apply.run_wall_time_us":1731,"num_ops":2,"prepare.queue_time_us":9107,"prepare.run_cpu_time_us":1862,"prepare.run_wall_time_us":30302,"raft.queue_time_us":66,"raft.run_cpu_time_us":477,"raft.run_wall_time_us":477,"replication_time_us":8080}]]}
I20250110 20:57:19.202379 4898 master_service.cc:946] ReplaceTablet: received request to replace tablet da6a9a054aa54455b4979e4c633ad6b4 from {username='slave'} at 127.0.0.1:36568
W20250110 20:57:19.151013 4899 rpcz_store.cc:269] 0110 20:57:16.470115 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.470245 (+ 130us) service_pool.cc:225] Handling call
0110 20:57:19.132305 (+2662060us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:17.562981 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:17.563327 (+ 346us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:17.563349 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:17.563363 (+ 14us) tablet.cc:659] Decoding operations
0110 20:57:17.563599 (+ 236us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:17.563632 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:17.563653 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:17.563908 (+ 255us) tablet.cc:698] Row locks acquired
0110 20:57:17.563921 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:17.564024 (+ 103us) write_op.cc:270] Start()
0110 20:57:17.564101 (+ 77us) write_op.cc:276] Timestamp: P: 1736542637563936 usec, L: 0
0110 20:57:17.564115 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:17.564549 (+ 434us) log.cc:844] Serialized 533 byte log entry
0110 20:57:17.595594 (+ 31045us) op_driver.cc:464] REPLICATION: finished
0110 20:57:17.596573 (+ 979us) write_op.cc:301] APPLY: starting
0110 20:57:17.596623 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:17.596699 (+ 76us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:17.596790 (+ 91us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:17.596943 (+ 153us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:17.597061 (+ 118us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:17.597099 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:57:17.597242 (+ 143us) log.cc:844] Serialized 23 byte log entry
0110 20:57:17.597371 (+ 129us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.597574 (+ 203us) write_op.cc:454] Released schema lock
0110 20:57:17.597603 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:17.597789 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:17.597833 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":208,"reported_tablets":27,"spinlock_wait_cycles":1408,"child_traces":[["op",{"apply.queue_time_us":850,"apply.run_cpu_time_us":1088,"apply.run_wall_time_us":1187,"num_ops":1,"prepare.queue_time_us":25872,"prepare.run_cpu_time_us":1635,"prepare.run_wall_time_us":1734,"replication_time_us":31409}]]}
I20250110 20:57:19.204937 5597 raft_consensus.cc:357] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:19.201144 4900 rpcz_store.cc:269] 0110 20:57:15.453512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.866840 (+413328us) service_pool.cc:225] Handling call
0110 20:57:16.974512 (+1107672us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:16.974536 (+ 24us) write_op.cc:454] Released schema lock
0110 20:57:19.172841 (+2198305us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:16.951773 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:16.952107 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:16.952132 (+ 25us) write_op.cc:435] Acquired schema lock
0110 20:57:16.952145 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:16.952398 (+ 253us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:16.952437 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:16.952458 (+ 21us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:16.952584 (+ 126us) tablet.cc:698] Row locks acquired
0110 20:57:16.952597 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:16.952700 (+ 103us) write_op.cc:270] Start()
0110 20:57:16.952777 (+ 77us) write_op.cc:276] Timestamp: P: 1736542636952628 usec, L: 0
0110 20:57:16.952791 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:16.953237 (+ 446us) log.cc:844] Serialized 533 byte log entry
0110 20:57:16.960374 (+ 7137us) op_driver.cc:464] REPLICATION: finished
0110 20:57:16.961791 (+ 1417us) write_op.cc:301] APPLY: starting
0110 20:57:16.961840 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:16.961930 (+ 90us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:16.962010 (+ 80us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:16.962159 (+ 149us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:16.962286 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:16.962326 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:16.962466 (+ 140us) log.cc:844] Serialized 23 byte log entry
0110 20:57:16.962582 (+ 116us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:16.962639 (+ 57us) write_op.cc:454] Released schema lock
0110 20:57:16.962664 (+ 25us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":21812,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":1149,"apply.run_cpu_time_us":1042,"apply.run_wall_time_us":1160,"num_ops":1,"prepare.queue_time_us":30031,"prepare.run_cpu_time_us":1746,"prepare.run_wall_time_us":1746,"raft.queue_time_us":1155,"raft.run_cpu_time_us":616,"raft.run_wall_time_us":13108,"replication_time_us":7494}]]}
I20250110 20:57:19.208213 5597 raft_consensus.cc:383] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:19.209815 5597 raft_consensus.cc:738] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
W20250110 20:57:19.210456 5110 consensus_peers.cc:487] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 -> Peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Couldn't send request to peer 3c5bff1e70804ab4a1bb63bdb9345ce6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527bbddda8184a6eb67c2411ae50db63. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:19.206195 5144 tablet_service.cc:1467] Processing CreateTablet for tablet e324e1bdd5754864a0f65c50594c2d93 (DEFAULT_TABLE table=default.Table_01d2ed1a594948c4995e0b934e4d2860 [id=e9f106d1b737444baaea6e97a8a3884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.211692 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e324e1bdd5754864a0f65c50594c2d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.210412 5597 consensus_queue.cc:260] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.216140 6059 raft_consensus.cc:491] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:19.223477 6059 raft_consensus.cc:513] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.225720 6059 leader_election.cc:290] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:19.234002 5597 ts_tablet_manager.cc:1428] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.034s user 0.007s sys 0.000s
I20250110 20:57:19.235128 5597 tablet_bootstrap.cc:492] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:18.958539 6205 consensus_queue.cc:1035] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:19.243535 6175 raft_consensus.cc:491] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:19.244073 6175 raft_consensus.cc:513] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.246332 6175 leader_election.cc:290] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:18.972666 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57941aee12a4ebea2b47f4727fc6eee" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:18.979616 5438 tablet_service.cc:1467] Processing CreateTablet for tablet 937831ec98d74007b8207ecb30843550 (DEFAULT_TABLE table=default.Table_4a1acf754c984e86ac40fd7ab1509b2b [id=33ffc109fdf34bc1b9de49832d42bee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.247421 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 937831ec98d74007b8207ecb30843550. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.247964 5597 tablet_bootstrap.cc:654] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:18.985220 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab3f2a8655c4f299f3fc62d11fad439 (DEFAULT_TABLE table=default.Table_bdd0ff0e681b4f57ba64a67252186056 [id=65c1840b0b34455f8d0c67e58b795263]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.249970 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab3f2a8655c4f299f3fc62d11fad439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:18.992838 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 5bde9f66fa4647558d50e1b2febd4335 (DEFAULT_TABLE table=default.Table_a262069587954407a948374ed4c1b18f [id=125620153cf443888a18ce75d09c8649]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.002777 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 97dfd451501d49909b3d163a53548ce4 (DEFAULT_TABLE table=default.Table_99db920d921840a19db98cc103c4fdab [id=1c212ce6a4d8478987cffb863cafb48b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.251591 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97dfd451501d49909b3d163a53548ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.012688 5439 tablet_service.cc:1467] Processing CreateTablet for tablet 36bca826ec144f77b96b4a41277fd6d5 (DEFAULT_TABLE table=default.Table_c7eb26eb1ef541e7853e3445290e6d49 [id=d8003ca561f74cbda31f38b6e9c51f6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.253938 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bca826ec144f77b96b4a41277fd6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.041954 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de354807004e26bfb147f1d420037e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.042776 5443 tablet_service.cc:1467] Processing CreateTablet for tablet d2b789aad72643149da01b9a607e2ce5 (DEFAULT_TABLE table=default.Table_754923f444f045e9b7d6c41c1b11bef0 [id=fa79e948d9db4cc89c8845b76c1c4e5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.257983 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet c6914e258314423c859eb31e85853274 with delete_type TABLET_DATA_DELETED (replaced by tablet a687769f69d844cc8564e61dc324d2b2) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.258164 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b789aad72643149da01b9a607e2ce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.258539 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet e361fb8b47d74548b5a2326d25a9e28c with delete_type TABLET_DATA_DELETED (replaced by tablet 491626848ad54be3a18186c65d6dbb60) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.259083 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 2967c06715e241dcbbe784eed2373704 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.259591 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c1ca1670916401ab0a43685df1b6759 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.260082 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cf0c87159574bfcbc7a34abe508f9d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.260592 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet 527bbddda8184a6eb67c2411ae50db63 with delete_type TABLET_DATA_DELETED (replaced by tablet 9633eb4580c04f11a48ff5055e257c50) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.261102 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet f0b5ba9f38c1493bba01180744f080e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:07 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.261675 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bde9f66fa4647558d50e1b2febd4335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.043710 5437 tablet_service.cc:1467] Processing CreateTablet for tablet bf4fbb2af0b84d35bd279ea2d5dd4ede (DEFAULT_TABLE table=default.Table_68433248a92d4f90bf5f613e7722a04c [id=edc9d679d92b49129d6ad5e695d96b13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.044121 5434 tablet_service.cc:1467] Processing CreateTablet for tablet ac9a20eb4dcb40d7b2e1dd1e5ed2edbb (DEFAULT_TABLE table=default.Table_43a94d9dc0074febbfe6afa4d4c9716d [id=0387c31491b6494580f9b02da7b0b022]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.045006 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 017a93be79ad49f0a72857cae4ada768 (DEFAULT_TABLE table=default.Table_36d9c90774f84119abe713d7b6331a49 [id=1441f7755521427390ee18c201f2421b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.050041 5432 tablet_service.cc:1467] Processing CreateTablet for tablet b5becf70b58847559b70ce27edc30398 (DEFAULT_TABLE table=default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.058132 5433 tablet_service.cc:1467] Processing CreateTablet for tablet 61d67ac398624f05b5a51e61d25e3e35 (DEFAULT_TABLE table=default.Table_f21a12015eab4635890b871de037c960 [id=3498f91140e34fb9b5ff861c40522ba3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.059535 5435 tablet_service.cc:1467] Processing CreateTablet for tablet 4d9672dc9cbb4e27bf6b01f513b6ccb8 (DEFAULT_TABLE table=default.Table_82e8a56b6f35429a9996b1622048bf8f [id=6e78cec050ce42a995a1cbb58325a4fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.069023 5431 tablet_service.cc:1467] Processing CreateTablet for tablet d26c2bd25161489baa156e78128e35aa (DEFAULT_TABLE table=default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.076642 5430 tablet_service.cc:1467] Processing CreateTablet for tablet e3e8cca02bf54c5086ecc8562c354d18 (DEFAULT_TABLE table=default.Table_b26866e23984441f96358c0d662acea3 [id=37073a31306f45e9b0e073cecbf1d05d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.086366 5429 tablet_service.cc:1467] Processing CreateTablet for tablet 873b9a1c522d4f7895740db18a537a1b (DEFAULT_TABLE table=default.Table_504da40a6cc54bb4b377ba25e145e4e4 [id=c450ef62b2b44988b1162f0bb99b260f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.097121 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 55542f8735d141789205e602651d38c6 (DEFAULT_TABLE table=default.Table_1e56771d3584418e8e38ebbff845ad4b [id=a3ecf6fce8db49cca87185958657ad9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.099395 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 4d0cee6de1f94451ba3530f684226055 (DEFAULT_TABLE table=default.Table_e7d42f471e7a4c3999d9b4ed548ae2f6 [id=79d7e17217e74200867a6656410916b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.115612 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9c229f569c4468bf43e7e880f7f0df (DEFAULT_TABLE table=default.Table_e3046e912cd243e591470e603e86c4d5 [id=81cce18d8d0e4c5fbb6dc13bc244cfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.119415 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:06 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.264006 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4fbb2af0b84d35bd279ea2d5dd4ede. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.264334 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 491626848ad54be3a18186c65d6dbb60 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.264874 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:08 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.265395 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc7b1acf4114bf69d801437855760df with delete_type TABLET_DATA_DELETED (replaced by tablet 5efe92b10aea46608d55d5e5ac36eca1) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.265909 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.266433 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 637a39e4264b40dd9480dc13e37359d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.266907 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a520230d9fd4a74914e46f784dcf747 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:05 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.267400 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet da24904dece04a6194790054f6d7153a with delete_type TABLET_DATA_DELETED (replaced by tablet bd9d0249798e4f24adf0b36d9e059b7e) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:19.185034 5399 consensus_peers.cc:487] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:19.268738 5399 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 83805.5us
I20250110 20:57:19.227084 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637a39e4264b40dd9480dc13e37359d9" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:19.270207 5463 raft_consensus.cc:2463] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:19.270656 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet e27e43461251413b97721323cd935374 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.271200 5440 tablet_service.cc:1514] Processing DeleteTablet for tablet d1afe41a09954fc8873ea53a330ea03f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:13 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.272192 5110 leader_election.cc:304] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:19.272173 5440 tablet_service.cc:1467] Processing CreateTablet for tablet e324e1bdd5754864a0f65c50594c2d93 (DEFAULT_TABLE table=default.Table_01d2ed1a594948c4995e0b934e4d2860 [id=e9f106d1b737444baaea6e97a8a3884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.272933 6059 raft_consensus.cc:2798] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:19.274089 5427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0cee6de1f94451ba3530f684226055. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.277984 5426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9c229f569c4468bf43e7e880f7f0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.281844 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 017a93be79ad49f0a72857cae4ada768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.282419 5430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e8cca02bf54c5086ecc8562c354d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.273241 6059 raft_consensus.cc:491] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:19.282752 6059 raft_consensus.cc:3054] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:19.285924 5431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26c2bd25161489baa156e78128e35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.290057 5433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d67ac398624f05b5a51e61d25e3e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.294473 5432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5becf70b58847559b70ce27edc30398. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.297823 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9a20eb4dcb40d7b2e1dd1e5ed2edbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.302058 5435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9672dc9cbb4e27bf6b01f513b6ccb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.267876 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d6eca349514b9fb0fff7cd6230027c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.302582 5428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55542f8735d141789205e602651d38c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.303200 5429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873b9a1c522d4f7895740db18a537a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.303768 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e324e1bdd5754864a0f65c50594c2d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.297135 6059 raft_consensus.cc:513] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.330624 6269 raft_consensus.cc:491] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:19.331100 6269 raft_consensus.cc:513] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.332989 6269 leader_election.cc:290] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:19.332950 4895 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e7f52abe4b384cb2abf0e1b9ee4abdc0" } modify_external_catalogs: true
I20250110 20:57:19.333465 4895 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e7f52abe4b384cb2abf0e1b9ee4abdc0" } modify_external_catalogs: true
I20250110 20:57:19.337383 6059 leader_election.cc:290] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:19.340381 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:19.342512 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "637a39e4264b40dd9480dc13e37359d9" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:19.343145 5460 raft_consensus.cc:3054] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:19.339118 5594 raft_consensus.cc:383] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:19.358471 5594 raft_consensus.cc:738] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:19.360991 4898 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet da6a9a054aa54455b4979e4c633ad6b4
I20250110 20:57:19.362535 4898 catalog_manager.cc:6596] ReplaceTablet: tablet da6a9a054aa54455b4979e4c633ad6b4 deleted and replaced by tablet a97def2f16e94061b1f938f73465fd31
I20250110 20:57:19.367398 5289 tablet_service.cc:1467] Processing CreateTablet for tablet 0e9c229f569c4468bf43e7e880f7f0df (DEFAULT_TABLE table=default.Table_e3046e912cd243e591470e603e86c4d5 [id=81cce18d8d0e4c5fbb6dc13bc244cfa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.368541 5289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9c229f569c4468bf43e7e880f7f0df. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.385649 5288 tablet_service.cc:1514] Processing DeleteTablet for tablet d6176c29f01b4ca5888c844c131270f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 700374c672fe43c9a45f34d0255b8683) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.385592 4897 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_6620f092e4b4461a8038cb09f49b2f81" } new_table_name: "default.Table_fd872d12446c4621a0aaad2289d63e1d" modify_external_catalogs: true
I20250110 20:57:19.374926 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet da6a9a054aa54455b4979e4c633ad6b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a97def2f16e94061b1f938f73465fd31) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.388545 5288 tablet_service.cc:1514] Processing DeleteTablet for tablet c57941aee12a4ebea2b47f4727fc6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.389029 4895 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 0c777c5e4fa74953a2796cbcf91e713b
I20250110 20:57:19.403755 5300 tablet_service.cc:1514] Processing DeleteTablet for tablet e508061e93cc430cb88f6d10d9684f00 with delete_type TABLET_DATA_DELETED (replaced by tablet e324e1bdd5754864a0f65c50594c2d93) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.405486 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ab3f2a8655c4f299f3fc62d11fad439. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.406033 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bde9f66fa4647558d50e1b2febd4335. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.414976 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97dfd451501d49909b3d163a53548ce4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.417933 5147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36bca826ec144f77b96b4a41277fd6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.420344 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d5c46de95bc4227bffc5e8425b8cc95" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
W20250110 20:57:19.421026 5257 leader_election.cc:343] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:19.421849 5111 leader_election.cc:304] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:19.422677 5146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2b789aad72643149da01b9a607e2ce5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.422895 5111 consensus_peers.cc:487] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6914e258314423c859eb31e85853274. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:19.423431 5111 consensus_peers.cc:487] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 527bbddda8184a6eb67c2411ae50db63. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:19.423826 5111 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 637a39e4264b40dd9480dc13e37359d9
I20250110 20:57:19.424528 5111 leader_election.cc:304] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:19.426014 5288 tablet_service.cc:1514] Processing DeleteTablet for tablet bae52ea080f54d228b6b76922248ed5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.426493 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf4fbb2af0b84d35bd279ea2d5dd4ede. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.426474 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet fde470279464496aa7efdee3800754f3 because the tablet was not found. No further retry: Not found: Tablet not found: fde470279464496aa7efdee3800754f3
I20250110 20:57:19.426752 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c87076a8a4b1e86f7e4844e17b1f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:19.427299 4886 catalog_manager.cc:4945] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): alter failed for tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]),no further retry: Not found: Tablet not found: d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:19.427872 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb75e1886e74a719dc09ccad8e1e284" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:19.428917 5287 tablet_service.cc:1514] Processing DeleteTablet for tablet da6a9a054aa54455b4979e4c633ad6b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a97def2f16e94061b1f938f73465fd31) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.429912 6269 raft_consensus.cc:2743] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:19.430397 6269 raft_consensus.cc:2743] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:19.431460 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 017a93be79ad49f0a72857cae4ada768. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.432433 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5becf70b58847559b70ce27edc30398. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.432984 5255 leader_election.cc:343] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:19.433331 5255 leader_election.cc:304] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:19.433537 4900 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e50ee362eaec42db8d62723516fded43" } modify_external_catalogs: true
I20250110 20:57:19.434026 4900 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e50ee362eaec42db8d62723516fded43" } modify_external_catalogs: true
I20250110 20:57:19.434860 5300 tablet_service.cc:1467] Processing CreateTablet for tablet e324e1bdd5754864a0f65c50594c2d93 (DEFAULT_TABLE table=default.Table_01d2ed1a594948c4995e0b934e4d2860 [id=e9f106d1b737444baaea6e97a8a3884a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.435807 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e324e1bdd5754864a0f65c50594c2d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.437932 5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61d67ac398624f05b5a51e61d25e3e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.440290 6358 raft_consensus.cc:2743] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:19.440791 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac9a20eb4dcb40d7b2e1dd1e5ed2edbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.453949 5143 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9672dc9cbb4e27bf6b01f513b6ccb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.458379 5595 ts_tablet_manager.cc:1918] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:19.458823 5595 log.cc:1198] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/41de354807004e26bfb147f1d420037e
I20250110 20:57:19.459663 5595 ts_tablet_manager.cc:1939] T 41de354807004e26bfb147f1d420037e P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:19.387586 5594 consensus_queue.cc:260] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:19.467181 5148 tablet_service.cc:1514] Processing DeleteTablet for tablet da6a9a054aa54455b4979e4c633ad6b4 with delete_type TABLET_DATA_DELETED (replaced by tablet a97def2f16e94061b1f938f73465fd31) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:19.471812 4891 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 3d1d5fba23d8411aa7cbf4781f1c7fc1 (table default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145]). Expected version 1 got 0
I20250110 20:57:19.475994 5592 ts_tablet_manager.cc:1918] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:19.476456 5592 log.cc:1198] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/cd0a666892d04e97a4b9967e38ab0946
I20250110 20:57:19.477188 5592 ts_tablet_manager.cc:1939] T cd0a666892d04e97a4b9967e38ab0946 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:19.477931 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 199) took 3717 ms. Trace:
I20250110 20:57:19.478261 5592 rpcz_store.cc:276] 0110 20:57:15.760524 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.760728 (+ 204us) service_pool.cc:225] Handling call
0110 20:57:18.581847 (+2821119us) ts_tablet_manager.cc:1145] Deleting tablet cd0a666892d04e97a4b9967e38ab0946
0110 20:57:18.581883 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.630798 (+ 48915us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b54000f1254. stack: 00007f2677942853 00007f2677942750 00007f26774b7584 00000000004fcd24 00000000004fccb9 00007f267e8426d1 00007f267e8c5352 00007f267ece21ac 00007f267f8f2d5a 00007f267f922ca0 00007f267f909e6d 00007f267f909e19 00007f267f909da9 00007f267f909d71 00007f267f908f9d 000000000059ea04
0110 20:57:18.836049 (+205251us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:19.475780 (+639731us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:19.477880 (+ 2100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98060544,"tablet-delete.queue_time_us":2752996}
I20250110 20:57:19.479579 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 200) took 3705 ms. Trace:
I20250110 20:57:19.503340 5595 tablet_replica.cc:331] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:19.507946 5597 tablet_bootstrap.cc:492] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:19.508761 5597 ts_tablet_manager.cc:1397] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.274s user 0.025s sys 0.057s
W20250110 20:57:19.480685 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7df3aafea18f49249cceb8a9431064e6 consensus state has no leader
I20250110 20:57:19.496083 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet cd0a666892d04e97a4b9967e38ab0946 (table default.Table_c3b44e7a68ac47a1ad0f7c5496130418 [id=143a11fac7564c92b6a7224e6ac31bfb]) successfully deleted
I20250110 20:57:19.503376 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f5aec17b06764886ad9b70bf235a2da0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.525617 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5aec17b06764886ad9b70bf235a2da0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:19.512070 4897 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a68d4b44975b436fb1b4a9ee6ff290d2 consensus state has no leader
I20250110 20:57:19.541630 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_2dfaee1ffeab4425896fad5e23dbb691"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.543035 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dfaee1ffeab4425896fad5e23dbb691 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.544673 4893 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_d67d43097b044b11bff93e2e95a7922a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.545960 4893 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67d43097b044b11bff93e2e95a7922a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.558362 4900 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 088879328f864f249bf8bba7749597c9
I20250110 20:57:19.475134 5303 tablet_service.cc:1467] Processing CreateTablet for tablet d26c2bd25161489baa156e78128e35aa (DEFAULT_TABLE table=default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.578943 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26c2bd25161489baa156e78128e35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.581892 5595 raft_consensus.cc:2238] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:19.468695 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 873b9a1c522d4f7895740db18a537a1b (DEFAULT_TABLE table=default.Table_504da40a6cc54bb4b377ba25e145e4e4 [id=c450ef62b2b44988b1162f0bb99b260f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.582482 5595 raft_consensus.cc:2267] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:19.582918 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873b9a1c522d4f7895740db18a537a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.598363 5597 raft_consensus.cc:357] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.599025 5597 raft_consensus.cc:383] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:19.599258 5597 raft_consensus.cc:738] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:19.601715 5142 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d26c2bd25161489baa156e78128e35aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.602358 5597 consensus_queue.cc:260] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:19.603482 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 178994us
I20250110 20:57:19.606426 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_652379995947455b9a395736e35ca034"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.607801 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_652379995947455b9a395736e35ca034 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.610105 4898 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_66d144e5d786435c8a4a7a14638914ac" } modify_external_catalogs: true
I20250110 20:57:19.610621 5141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3e8cca02bf54c5086ecc8562c354d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.610558 4898 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_66d144e5d786435c8a4a7a14638914ac" } modify_external_catalogs: true
I20250110 20:57:19.611151 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 223) took 2492 ms. Trace:
I20250110 20:57:19.611438 5773 rpcz_store.cc:276] 0110 20:57:17.118897 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.119014 (+ 117us) service_pool.cc:225] Handling call
0110 20:57:18.695849 (+1576835us) ts_tablet_manager.cc:1145] Deleting tablet 01962c5b7ae64f80b998948d18250b4c
0110 20:57:18.695923 (+ 74us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:18.905085 (+209162us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:18.986049 (+ 80964us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:19.611097 (+625048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576041}
I20250110 20:57:19.612825 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 01962c5b7ae64f80b998948d18250b4c (table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]) successfully deleted
I20250110 20:57:19.589025 6269 consensus_queue.cc:1035] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20250110 20:57:19.486577 5594 ts_tablet_manager.cc:1428] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.536s user 0.004s sys 0.001s
I20250110 20:57:19.496601 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb75e1886e74a719dc09ccad8e1e284" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:19.639602 5594 ts_tablet_manager.cc:1460] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6: Tablet startup took 688ms
W20250110 20:57:19.639884 5594 ts_tablet_manager.cc:1462] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trace:
0110 20:57:10.022759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.097565 (+ 74806us) service_pool.cc:225] Handling call
0110 20:57:10.098951 (+ 1386us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:10.099042 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:10.205586 (+106544us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.223456 (+ 17870us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:10.223850 (+ 394us) inbound_call.cc:173] Queueing success response
0110 20:57:18.480482 (+8256632us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0110 20:57:18.950922 (+470440us) ts_tablet_manager.cc:1429] Starting tablet replica
0110 20:57:18.951415 (+ 493us) tablet_replica.cc:260] Starting instrumentation
0110 20:57:18.951834 (+ 419us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":3091,"raft.queue_time_us":7684,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":24403,"tablet-open.queue_time_us":8256733}
I20250110 20:57:19.497368 6355 raft_consensus.cc:491] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:19.497761 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c87076a8a4b1e86f7e4844e17b1f3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:19.571563 6300 consensus_queue.cc:1035] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:57:19.641602 6355 raft_consensus.cc:513] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:19.643457 6355 leader_election.cc:290] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:19.646229 5594 tablet_bootstrap.cc:492] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:19.651940 5594 tablet_bootstrap.cc:654] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:19.656534 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:19.657115 5320 raft_consensus.cc:2463] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:19.657522 6175 raft_consensus.cc:491] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:19.657892 6175 raft_consensus.cc:513] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:19.658351 5398 leader_election.cc:304] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:19.659210 6355 raft_consensus.cc:2798] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:19.659509 6355 raft_consensus.cc:491] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:19.659704 6175 leader_election.cc:290] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:19.659780 6355 raft_consensus.cc:3054] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:19.662081 5398 consensus_peers.cc:487] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 41de354807004e26bfb147f1d420037e. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:19.665256 5595 ts_tablet_manager.cc:1905] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:19.673061 5600 tablet_bootstrap.cc:492] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:19.674795 5597 ts_tablet_manager.cc:1428] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.165s user 0.004s sys 0.000s
I20250110 20:57:19.676438 5600 ts_tablet_manager.cc:1397] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.833s user 0.010s sys 0.000s
I20250110 20:57:19.678607 6269 consensus_queue.cc:1035] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:19.687489 5592 rpcz_store.cc:276] 0110 20:57:15.773926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.774141 (+ 215us) service_pool.cc:225] Handling call
0110 20:57:19.479363 (+3705222us) ts_tablet_manager.cc:1145] Deleting tablet d310cf52aa8c4930a7821117a46dc958
0110 20:57:19.479398 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:19.479536 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3650288}
I20250110 20:57:19.688654 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]) successfully deleted
W20250110 20:57:19.693076 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 637a39e4264b40dd9480dc13e37359d9 because the tablet was not found. No further retry: Not found: Tablet not found: 637a39e4264b40dd9480dc13e37359d9
W20250110 20:57:19.699122 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet e27e43461251413b97721323cd935374 because the tablet was not found. No further retry: Not found: Tablet not found: e27e43461251413b97721323cd935374
W20250110 20:57:19.700058 5257 leader_election.cc:343] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:19.701447 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet c44ca47c747544b6afec83b399dcdcb5 because the tablet was not found. No further retry: Not found: Tablet not found: c44ca47c747544b6afec83b399dcdcb5
W20250110 20:57:19.703094 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet d310cf52aa8c4930a7821117a46dc958 because the tablet was not found. No further retry: Not found: Tablet not found: d310cf52aa8c4930a7821117a46dc958
I20250110 20:57:19.703950 5592 tablet_replica.cc:331] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:19.704994 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 7462d21b91c2441fb9e11383466e9ca7 because the tablet was not found. No further retry: Not found: Tablet not found: 7462d21b91c2441fb9e11383466e9ca7
W20250110 20:57:19.709550 5257 leader_election.cc:343] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:19.708210 5458 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:19.716017 5592 raft_consensus.cc:2238] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:19.716845 5592 raft_consensus.cc:2267] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:19.718465 5592 ts_tablet_manager.cc:1905] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:19.732038 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:19.711314 5600 raft_consensus.cc:357] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.732868 5600 raft_consensus.cc:383] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:19.733108 5600 raft_consensus.cc:738] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:19.751873 5140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 873b9a1c522d4f7895740db18a537a1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.756297 5139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55542f8735d141789205e602651d38c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.761526 5138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d0cee6de1f94451ba3530f684226055. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.765957 5137 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9c229f569c4468bf43e7e880f7f0df. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.766166 5255 leader_election.cc:343] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:19.766523 5255 leader_election.cc:304] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:19.767093 6175 raft_consensus.cc:2743] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:19.767235 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a23e7391e6744a67a193c12dc0ca20c5" } new_table_name: "default.Table_fd1dd024f4824116bfc0ceb0584e3f8a" modify_external_catalogs: true
W20250110 20:57:19.768007 5255 leader_election.cc:343] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:19.768313 5255 leader_election.cc:304] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:19.768859 6175 raft_consensus.cc:2743] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:19.770030 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_f52be800c137445c9aaf68c8654c80b2" } new_table_name: "default.Table_c8a405e3642b40e994712f0076f84e63" modify_external_catalogs: true
W20250110 20:57:19.773203 6305 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 380676555299961068) tablet 00000000000000000000000000000000 cur holder 0x7b5800070e00
I20250110 20:57:19.774945 4897 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a0d2f2c0efd3421980eae4849cea168a" } modify_external_catalogs: true
I20250110 20:57:19.775439 4897 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a0d2f2c0efd3421980eae4849cea168a" } modify_external_catalogs: true
I20250110 20:57:19.781812 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 227) took 2445 ms. Trace:
I20250110 20:57:19.782141 5773 rpcz_store.cc:276] 0110 20:57:17.336598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.338627 (+ 2029us) service_pool.cc:225] Handling call
0110 20:57:19.612379 (+2273752us) ts_tablet_manager.cc:1145] Deleting tablet c7961f8bd3ff42ffab214a04c704fd26
0110 20:57:19.746162 (+133783us) spinlock_profiling.cc:243] Waited 133 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c98c2 00007f896525b999 00007f89644c05f8 00007f89645843e9 00007f89645849e3 00007f89645b4ca0 00007f896459be6d 00007f896459be19 00007f896459bda9
0110 20:57:19.781481 (+ 35319us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:19.781757 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":305113856,"tablet-delete.queue_time_us":2273244}
W20250110 20:57:19.784526 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet c7961f8bd3ff42ffab214a04c704fd26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7961f8bd3ff42ffab214a04c704fd26 already in progress: creating tablet
I20250110 20:57:19.785578 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 235) took 1766 ms. Trace:
I20250110 20:57:19.785959 5773 rpcz_store.cc:276] 0110 20:57:18.019126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.019224 (+ 98us) service_pool.cc:225] Handling call
0110 20:57:19.785296 (+1766072us) ts_tablet_manager.cc:1145] Deleting tablet 1ac07ac6fc9f477ba7f9accc5533be39
0110 20:57:19.785339 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:19.785522 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1753533}
W20250110 20:57:19.787247 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 1ac07ac6fc9f477ba7f9accc5533be39 because the tablet was not found. No further retry: Not found: Tablet not found: 1ac07ac6fc9f477ba7f9accc5533be39
I20250110 20:57:19.796844 5773 tablet_replica.cc:331] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:19.733703 5600 consensus_queue.cc:260] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:19.797628 5773 raft_consensus.cc:2238] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:19.798326 5773 raft_consensus.cc:2267] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:19.801262 5600 ts_tablet_manager.cc:1428] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.115s user 0.005s sys 0.000s
I20250110 20:57:19.801952 5435 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2f296aca449f28b968f9559d8eef4 (DEFAULT_TABLE table=default.Table_b3b9ceb781ee4ac0ac6c046d253749c7 [id=3988de6dc5764bdf977027c61f2a9f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.802965 5435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b2f296aca449f28b968f9559d8eef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.811229 4895 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_6200af64376b455fac5a660790cd0e90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.812616 4895 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6200af64376b455fac5a660790cd0e90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.814308 5600 tablet_bootstrap.cc:492] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:19.815268 4892 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:19.815282 5431 tablet_service.cc:1467] Processing CreateTablet for tablet b8b6e9a462404455a10ff50898f67b1c (DEFAULT_TABLE table=default.Table_c8c39c1a1dde4e46a4f0aaabe3045c0d [id=f08beadf7b5343958f3da924909f2ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.816295 5431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b6e9a462404455a10ff50898f67b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.816289 4892 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 5eb75e1886e74a719dc09ccad8e1e284 (table default.Table_5f3f989b2ecf48a0a72e17cc18a2b095 [id=76a6826d1a6b46bf8b300259c63eb1b9]). Expected version 2 got 0
I20250110 20:57:19.816793 4894 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_0829957c7240427fb7f1fa27fc93cf1c [id=7171d9e7286c4a7caf0423fd3d7d6145] alter complete (version 1)
I20250110 20:57:19.818107 5429 tablet_service.cc:1467] Processing CreateTablet for tablet a591ca10937f41d4af674c72b451ea6f (DEFAULT_TABLE table=default.Table_d569e0fc30134336bc26374a5eaeb7cf [id=55af54fde5d6442ab85013a3cea30f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.819046 5429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a591ca10937f41d4af674c72b451ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.822254 5426 tablet_service.cc:1467] Processing CreateTablet for tablet dd07a9eaa4364d1ca5b577a010978899 (DEFAULT_TABLE table=default.Table_66d144e5d786435c8a4a7a14638914ac [id=55807029b2104ca3a93974c1cfe3f4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.823184 5426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd07a9eaa4364d1ca5b577a010978899. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.814654 5257 leader_election.cc:343] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:19.834117 5432 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8f59800b2a49a1b69b43133d4d0551 (DEFAULT_TABLE table=default.Table_f52be800c137445c9aaf68c8654c80b2 [id=f4017feb5b0542a193d1353a818a7184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.835145 5432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8f59800b2a49a1b69b43133d4d0551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.838068 5433 tablet_service.cc:1467] Processing CreateTablet for tablet a2bcc237beae4d90aee701b0ee1ae14f (DEFAULT_TABLE table=default.Table_c1d432de9a1f4c218e0150fee53f6e6a [id=0db76fbc589f4a3bb700f06a52a66a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.839015 5433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bcc237beae4d90aee701b0ee1ae14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.842105 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 6437a175c6124504aff7665806cf05b8 (DEFAULT_TABLE table=default.Table_2167319943e24d52ad2939094f31e4c0 [id=a7f29c32518048c49832d6d298994b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.843042 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6437a175c6124504aff7665806cf05b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.848415 5430 tablet_service.cc:1467] Processing CreateTablet for tablet 700374c672fe43c9a45f34d0255b8683 (DEFAULT_TABLE table=default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.849370 5430 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700374c672fe43c9a45f34d0255b8683. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:19.851086 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6b8f59800b2a49a1b69b43133d4d0551 consensus state has no leader
W20250110 20:57:19.852025 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fdb2ee9419244a9ab939be56b2d7c72 consensus state has no leader
I20250110 20:57:19.863112 5289 tablet_service.cc:1467] Processing CreateTablet for tablet b8b6e9a462404455a10ff50898f67b1c (DEFAULT_TABLE table=default.Table_c8c39c1a1dde4e46a4f0aaabe3045c0d [id=f08beadf7b5343958f3da924909f2ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.864239 5289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b6e9a462404455a10ff50898f67b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.864238 6251 consensus_queue.cc:1035] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:19.874246 4893 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:19.877764 5298 tablet_service.cc:1467] Processing CreateTablet for tablet a591ca10937f41d4af674c72b451ea6f (DEFAULT_TABLE table=default.Table_d569e0fc30134336bc26374a5eaeb7cf [id=55af54fde5d6442ab85013a3cea30f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.877912 5291 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8f59800b2a49a1b69b43133d4d0551 (DEFAULT_TABLE table=default.Table_f52be800c137445c9aaf68c8654c80b2 [id=f4017feb5b0542a193d1353a818a7184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.878809 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a591ca10937f41d4af674c72b451ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.879676 5294 tablet_service.cc:1467] Processing CreateTablet for tablet a2bcc237beae4d90aee701b0ee1ae14f (DEFAULT_TABLE table=default.Table_c1d432de9a1f4c218e0150fee53f6e6a [id=0db76fbc589f4a3bb700f06a52a66a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.880549 5294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bcc237beae4d90aee701b0ee1ae14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.882728 5291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8f59800b2a49a1b69b43133d4d0551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.897940 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_fe05cd0420bb4aaa86a1e3789596e738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.899458 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe05cd0420bb4aaa86a1e3789596e738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.905594 4898 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet dd07a9eaa4364d1ca5b577a010978899
I20250110 20:57:19.908607 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_f982d5a7aa86433da097e579045584df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.910043 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f982d5a7aa86433da097e579045584df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.913349 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet dd07a9eaa4364d1ca5b577a010978899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:19 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.579906 5460 raft_consensus.cc:2463] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:19.728376 6356 raft_consensus.cc:491] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:19.736866 5594 tablet_bootstrap.cc:492] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:19.923185 5594 ts_tablet_manager.cc:1397] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.277s user 0.006s sys 0.005s
I20250110 20:57:19.931766 4897 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 5feb54cd239b4a80894d45898776fce5
I20250110 20:57:19.935321 5296 tablet_service.cc:1467] Processing CreateTablet for tablet dd07a9eaa4364d1ca5b577a010978899 (DEFAULT_TABLE table=default.Table_66d144e5d786435c8a4a7a14638914ac [id=55807029b2104ca3a93974c1cfe3f4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.936404 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd07a9eaa4364d1ca5b577a010978899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.942265 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 700374c672fe43c9a45f34d0255b8683 (DEFAULT_TABLE table=default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.943125 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700374c672fe43c9a45f34d0255b8683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.951413 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2f296aca449f28b968f9559d8eef4 (DEFAULT_TABLE table=default.Table_b3b9ceb781ee4ac0ac6c046d253749c7 [id=3988de6dc5764bdf977027c61f2a9f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.952368 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b2f296aca449f28b968f9559d8eef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.958911 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_6529554ce9fa48d482b0343488159c52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:19.960513 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6529554ce9fa48d482b0343488159c52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:19.964093 5597 tablet_bootstrap.cc:492] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:19.967486 5600 tablet_bootstrap.cc:654] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:19.922255 6356 raft_consensus.cc:513] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:19.977974 6356 leader_election.cc:290] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:19.989434 5398 leader_election.cc:343] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 48e53d95ef8645c0bd123138655305ac
I20250110 20:57:19.990886 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet 5feb54cd239b4a80894d45898776fce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:19 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:19.968192 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet dd07a9eaa4364d1ca5b577a010978899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:19 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:19.968879 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:20.011332 5156 tablet_service.cc:1467] Processing CreateTablet for tablet b8b6e9a462404455a10ff50898f67b1c (DEFAULT_TABLE table=default.Table_c8c39c1a1dde4e46a4f0aaabe3045c0d [id=f08beadf7b5343958f3da924909f2ad0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.012284 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b6e9a462404455a10ff50898f67b1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.012953 5154 tablet_service.cc:1467] Processing CreateTablet for tablet dd07a9eaa4364d1ca5b577a010978899 (DEFAULT_TABLE table=default.Table_66d144e5d786435c8a4a7a14638914ac [id=55807029b2104ca3a93974c1cfe3f4f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.970047 5290 tablet_service.cc:1514] Processing DeleteTablet for tablet 5feb54cd239b4a80894d45898776fce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:19 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:20.013891 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd07a9eaa4364d1ca5b577a010978899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.015838 5151 tablet_service.cc:1467] Processing CreateTablet for tablet a591ca10937f41d4af674c72b451ea6f (DEFAULT_TABLE table=default.Table_d569e0fc30134336bc26374a5eaeb7cf [id=55af54fde5d6442ab85013a3cea30f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.963688 6059 raft_consensus.cc:491] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.018882 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8f59800b2a49a1b69b43133d4d0551 (DEFAULT_TABLE table=default.Table_f52be800c137445c9aaf68c8654c80b2 [id=f4017feb5b0542a193d1353a818a7184]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.019698 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8f59800b2a49a1b69b43133d4d0551. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:19.973994 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 6437a175c6124504aff7665806cf05b8 (DEFAULT_TABLE table=default.Table_2167319943e24d52ad2939094f31e4c0 [id=a7f29c32518048c49832d6d298994b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.989264 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 41b2f296aca449f28b968f9559d8eef4 (DEFAULT_TABLE table=default.Table_b3b9ceb781ee4ac0ac6c046d253749c7 [id=3988de6dc5764bdf977027c61f2a9f06]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:19.990415 5595 ts_tablet_manager.cc:1918] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:20.021301 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a591ca10937f41d4af674c72b451ea6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.021504 5595 log.cc:1198] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f3104928188c489d93bfd499d148fa5d
I20250110 20:57:20.021718 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6437a175c6124504aff7665806cf05b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.022210 5595 ts_tablet_manager.cc:1939] T f3104928188c489d93bfd499d148fa5d P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:20.023559 5595 tablet_replica.cc:331] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:20.024246 5595 raft_consensus.cc:2238] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:20.024753 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet f3104928188c489d93bfd499d148fa5d (table default.Table_338aa5da81e146be96e0f8cc5dcae996 [id=75d13198e6a74915b9955b32e3908f11]) successfully deleted
I20250110 20:57:20.025106 5595 raft_consensus.cc:2267] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:20.025059 5148 tablet_service.cc:1467] Processing CreateTablet for tablet a2bcc237beae4d90aee701b0ee1ae14f (DEFAULT_TABLE table=default.Table_c1d432de9a1f4c218e0150fee53f6e6a [id=0db76fbc589f4a3bb700f06a52a66a86]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:20.025380 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet d6176c29f01b4ca5888c844c131270f2 because the tablet was not found. No further retry: Not found: Tablet not found: d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:20.025902 5148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2bcc237beae4d90aee701b0ee1ae14f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.026836 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b2f296aca449f28b968f9559d8eef4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.026975 5595 ts_tablet_manager.cc:1905] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:20.045377 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 700374c672fe43c9a45f34d0255b8683 (DEFAULT_TABLE table=default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.045666 5145 tablet_service.cc:1467] Processing CreateTablet for tablet 6437a175c6124504aff7665806cf05b8 (DEFAULT_TABLE table=default.Table_2167319943e24d52ad2939094f31e4c0 [id=a7f29c32518048c49832d6d298994b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.046303 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 700374c672fe43c9a45f34d0255b8683. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.047631 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6437a175c6124504aff7665806cf05b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.061946 5600 tablet_bootstrap.cc:492] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:20.069722 5600 ts_tablet_manager.cc:1397] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.256s user 0.010s sys 0.000s
I20250110 20:57:20.079770 4892 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_bdd0ff0e681b4f57ba64a67252186056" } new_table_name: "default.Table_31d9c425a797473bab523c3c7eb3165a" modify_external_catalogs: true
I20250110 20:57:20.082110 4900 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_f70bf3182e2a4bcbafbc98e2977b4f2d" } new_table_name: "default.Table_91e7b8f8b37f49ee8fc122bcec6530c0" modify_external_catalogs: true
I20250110 20:57:20.082250 5592 ts_tablet_manager.cc:1918] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:20.082643 5592 log.cc:1198] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/c951d5ca47e8409db646b43bb04a7034
I20250110 20:57:20.083256 5592 ts_tablet_manager.cc:1939] T c951d5ca47e8409db646b43bb04a7034 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:20.085018 5600 raft_consensus.cc:357] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.095381 5600 raft_consensus.cc:383] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:20.095691 5600 raft_consensus.cc:738] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:20.096400 5600 consensus_queue.cc:260] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.021965 6059 raft_consensus.cc:513] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.106961 6059 leader_election.cc:290] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:20.107698 5150 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 256) took 1070 ms. Trace:
I20250110 20:57:20.108055 5150 rpcz_store.cc:276] 0110 20:57:19.037417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.038641 (+ 1224us) service_pool.cc:225] Handling call
0110 20:57:19.431988 (+393347us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.432135 (+ 147us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.573815 (+141680us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:19.665196 (+ 91381us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.087558 (+422362us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c98c2 00007f896525b999 00007f89644c05f8 00007f89645840c3 00007f896457fe8f 00007f896458064d 00007f8964531859 00007f8962c19c64 00007f8962c19bf2
0110 20:57:20.107643 (+ 20085us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":302093184}
I20250110 20:57:20.109445 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c5b954dbccb45b2a6ab7cc5cceadffb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:11 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:20.115206 4895 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e3046e912cd243e591470e603e86c4d5" } modify_external_catalogs: true
I20250110 20:57:20.115741 4895 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_e3046e912cd243e591470e603e86c4d5" } modify_external_catalogs: true
I20250110 20:57:20.126240 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 637a39e4264b40dd9480dc13e37359d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:09 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:20.126822 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc7b1acf4114bf69d801437855760df with delete_type TABLET_DATA_DELETED (replaced by tablet 5efe92b10aea46608d55d5e5ac36eca1) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:20.127370 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet dd07a9eaa4364d1ca5b577a010978899 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:19 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:20.127992 5150 tablet_service.cc:1514] Processing DeleteTablet for tablet 5feb54cd239b4a80894d45898776fce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:19 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:20.134703 5600 ts_tablet_manager.cc:1428] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.061s user 0.002s sys 0.003s
I20250110 20:57:20.135576 5600 tablet_bootstrap.cc:492] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:20.136031 5155 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 254) took 1104 ms. Trace:
I20250110 20:57:20.136114 4893 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_01d2ed1a594948c4995e0b934e4d2860" } modify_external_catalogs: true
I20250110 20:57:20.136610 4893 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_01d2ed1a594948c4995e0b934e4d2860" } modify_external_catalogs: true
I20250110 20:57:20.136404 5155 rpcz_store.cc:276] 0110 20:57:19.031338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.031461 (+ 123us) service_pool.cc:225] Handling call
0110 20:57:19.429453 (+397992us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.429533 (+ 80us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.518861 (+ 89328us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:19.595679 (+ 76818us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.135769 (+540090us) spinlock_profiling.cc:243] Waited 82.8 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c98c2 00007f896525b999 00007f89644c05f8 00007f89645840c3 00007f896457fe8f 00007f896458064d 00007f8964531859 00007f8962c19c64 00007f8962c19bf2
0110 20:57:20.135985 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":190540032}
I20250110 20:57:20.138456 5149 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 257) took 1094 ms. Trace:
I20250110 20:57:20.138855 5149 rpcz_store.cc:276] 0110 20:57:19.044331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.044480 (+ 149us) service_pool.cc:225] Handling call
0110 20:57:19.437453 (+392973us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.437559 (+ 106us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.634739 (+197180us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.138023 (+503284us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.138400 (+ 377us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:20.154368 5600 tablet_bootstrap.cc:654] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:20.159477 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_fce3f8bf894845e287acf83b2e2fe8c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:20.160879 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fce3f8bf894845e287acf83b2e2fe8c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:20.161659 5773 ts_tablet_manager.cc:1905] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:20.167191 4886 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 51fae5f59d4642a09acee5a687b91717 (table default.Table_53bbb78e4859491ba4013f5dd868d449 [id=e4e3948de85c4db480e29af5b2d6c4e2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 51fae5f59d4642a09acee5a687b91717 consensus state has no leader
I20250110 20:57:20.191963 6175 raft_consensus.cc:491] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.192469 6175 raft_consensus.cc:513] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.194451 6175 leader_election.cc:290] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:20.197206 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_91296c9c7dfb47dab70fcfd591e7f6b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:20.198604 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91296c9c7dfb47dab70fcfd591e7f6b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:20.202006 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 2591beedb8d2456da1c282d667b156a9 (DEFAULT_TABLE table=default.Table_652379995947455b9a395736e35ca034 [id=01dddc8bb7fd43fabe5fa3860a152cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.202957 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2591beedb8d2456da1c282d667b156a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.206017 5300 tablet_service.cc:1467] Processing CreateTablet for tablet c27a964962af44c48f7fcfa9a130a08f (DEFAULT_TABLE table=default.Table_78b6d1b2ffe54b66aa5e067af6013c12 [id=5ce114353cb8462b833fd55d8258e355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.213977 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27a964962af44c48f7fcfa9a130a08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.214026 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c813130649cc4df1b483ac4a86b4f0a0" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:20.207494 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 948981b34bdd4017b7ce659e7f850c5f (DEFAULT_TABLE table=default.Table_f88b13f73f784b8890702e20bc102be2 [id=f99b7dcae8bb4a70b33f401a1cd1872e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.220638 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 948981b34bdd4017b7ce659e7f850c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.178112 5595 ts_tablet_manager.cc:1918] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:20.225554 5595 log.cc:1198] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/c57941aee12a4ebea2b47f4727fc6eee
I20250110 20:57:20.226258 5595 ts_tablet_manager.cc:1939] T c57941aee12a4ebea2b47f4727fc6eee P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
W20250110 20:57:20.226508 5255 leader_election.cc:343] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:20.228732 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 247) took 1094 ms. Trace:
I20250110 20:57:20.234186 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet c57941aee12a4ebea2b47f4727fc6eee (table default.Table_a0178b82e1d8457eb7f165640e222c72 [id=1a67e1de31b3490e8c391f502b5a02bb]) successfully deleted
I20250110 20:57:20.232581 5289 tablet_service.cc:1467] Processing CreateTablet for tablet c010af5c95c5421eb6bb7439f2164498 (DEFAULT_TABLE table=default.Table_d28fb676474445e5bc7377940f7a4c3a [id=9f2cc4681a9c447293b0218067fc3ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.239246 5289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c010af5c95c5421eb6bb7439f2164498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.239115 4897 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_2a752f83903946dea40ca65dd07107ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:20.239910 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 2591beedb8d2456da1c282d667b156a9 (DEFAULT_TABLE table=default.Table_652379995947455b9a395736e35ca034 [id=01dddc8bb7fd43fabe5fa3860a152cbe]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:20.240471 4897 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a752f83903946dea40ca65dd07107ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:20.233606 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 74fd501dc6284a2c9b1da666ee81f688 (DEFAULT_TABLE table=default.Table_2dfaee1ffeab4425896fad5e23dbb691 [id=36abbb2daa6648888440d2e9a579fef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.242400 5290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fd501dc6284a2c9b1da666ee81f688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.244751 5438 tablet_service.cc:1467] Processing CreateTablet for tablet c27a964962af44c48f7fcfa9a130a08f (DEFAULT_TABLE table=default.Table_78b6d1b2ffe54b66aa5e067af6013c12 [id=5ce114353cb8462b833fd55d8258e355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.233963 5301 tablet_service.cc:1467] Processing CreateTablet for tablet a97def2f16e94061b1f938f73465fd31 (DEFAULT_TABLE table=default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.245939 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97def2f16e94061b1f938f73465fd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.245930 5437 tablet_service.cc:1467] Processing CreateTablet for tablet 948981b34bdd4017b7ce659e7f850c5f (DEFAULT_TABLE table=default.Table_f88b13f73f784b8890702e20bc102be2 [id=f99b7dcae8bb4a70b33f401a1cd1872e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.247043 5441 tablet_service.cc:1467] Processing CreateTablet for tablet c010af5c95c5421eb6bb7439f2164498 (DEFAULT_TABLE table=default.Table_d28fb676474445e5bc7377940f7a4c3a [id=9f2cc4681a9c447293b0218067fc3ecb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.234822 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 210b8197624547d38c084def211b3244 (DEFAULT_TABLE table=default.Table_f5aec17b06764886ad9b70bf235a2da0 [id=37e03fb5b72a4d97b2564cb8f5dd8fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.249975 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 210b8197624547d38c084def211b3244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.251354 4891 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:20.252440 6175 raft_consensus.cc:491] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.235337 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 319c544143824575bd96ae9d808fbb9c (DEFAULT_TABLE table=default.Table_d67d43097b044b11bff93e2e95a7922a [id=15c3fc4ad0c140d399d1ef10a8f7d3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.252892 6175 raft_consensus.cc:513] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.253919 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319c544143824575bd96ae9d808fbb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.257992 6175 leader_election.cc:290] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:20.259145 4900 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb consensus state has no leader
W20250110 20:57:20.261375 4892 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ab3f2a8655c4f299f3fc62d11fad439 consensus state has no leader
I20250110 20:57:20.265208 5436 tablet_service.cc:1467] Processing CreateTablet for tablet 74fd501dc6284a2c9b1da666ee81f688 (DEFAULT_TABLE table=default.Table_2dfaee1ffeab4425896fad5e23dbb691 [id=36abbb2daa6648888440d2e9a579fef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.265496 5434 tablet_service.cc:1467] Processing CreateTablet for tablet 319c544143824575bd96ae9d808fbb9c (DEFAULT_TABLE table=default.Table_d67d43097b044b11bff93e2e95a7922a [id=15c3fc4ad0c140d399d1ef10a8f7d3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.265674 5439 tablet_service.cc:1467] Processing CreateTablet for tablet a97def2f16e94061b1f938f73465fd31 (DEFAULT_TABLE table=default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.267182 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 210b8197624547d38c084def211b3244 (DEFAULT_TABLE table=default.Table_f5aec17b06764886ad9b70bf235a2da0 [id=37e03fb5b72a4d97b2564cb8f5dd8fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.267896 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758430b311a8478c9cab2bcf79253b52" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:20.269996 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 202) took 4486 ms. Trace:
I20250110 20:57:20.270319 5592 rpcz_store.cc:276] 0110 20:57:15.783980 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.784085 (+ 105us) service_pool.cc:225] Handling call
0110 20:57:19.703806 (+3919721us) ts_tablet_manager.cc:1145] Deleting tablet c951d5ca47e8409db646b43bb04a7034
0110 20:57:19.703848 (+ 42us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:19.956392 (+252544us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.082035 (+125643us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.269941 (+187906us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3874602}
I20250110 20:57:20.271376 5592 tablet_replica.cc:331] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:20.277642 5592 raft_consensus.cc:2238] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:20.278870 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet c951d5ca47e8409db646b43bb04a7034 (table default.Table_6ca3c213749745caa228b62726499ecd [id=723ca40b46694de9802850cb89dca66f]) successfully deleted
W20250110 20:57:20.280891 5255 leader_election.cc:343] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:20.238461 5600 tablet_bootstrap.cc:492] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:20.282229 5600 ts_tablet_manager.cc:1397] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.147s user 0.006s sys 0.003s
I20250110 20:57:20.284385 5600 raft_consensus.cc:357] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.284768 5428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2591beedb8d2456da1c282d667b156a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.284993 5600 raft_consensus.cc:383] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:20.288805 5438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27a964962af44c48f7fcfa9a130a08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.289966 5437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 948981b34bdd4017b7ce659e7f850c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.294037 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c010af5c95c5421eb6bb7439f2164498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.294363 5595 rpcz_store.cc:276] 0110 20:57:19.134800 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.403603 (+268803us) service_pool.cc:225] Handling call
0110 20:57:20.228516 (+824913us) ts_tablet_manager.cc:1145] Deleting tablet e508061e93cc430cb88f6d10d9684f00
0110 20:57:20.228549 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:20.228689 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":824359}
W20250110 20:57:20.295519 4886 catalog_manager.cc:4839] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet e508061e93cc430cb88f6d10d9684f00 because the tablet was not found. No further retry: Not found: Tablet not found: e508061e93cc430cb88f6d10d9684f00
I20250110 20:57:20.296288 5595 tablet_replica.cc:331] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:20.296926 5595 raft_consensus.cc:2238] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:20.297662 5595 raft_consensus.cc:2267] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:20.298028 5436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fd501dc6284a2c9b1da666ee81f688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.299626 5595 ts_tablet_manager.cc:1905] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:20.301851 5434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319c544143824575bd96ae9d808fbb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.305853 5439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97def2f16e94061b1f938f73465fd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.309896 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 210b8197624547d38c084def211b3244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.319195 5592 raft_consensus.cc:2267] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:20.320971 5592 ts_tablet_manager.cc:1905] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:20.285229 5600 raft_consensus.cc:738] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:20.329973 5600 consensus_queue.cc:260] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.235494 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc7b1acf4114bf69d801437855760df" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:20.337460 5110 leader_election.cc:304] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:20.338657 5145 tablet_service.cc:1467] Processing CreateTablet for tablet c27a964962af44c48f7fcfa9a130a08f (DEFAULT_TABLE table=default.Table_78b6d1b2ffe54b66aa5e067af6013c12 [id=5ce114353cb8462b833fd55d8258e355]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.336213 5600 ts_tablet_manager.cc:1428] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20250110 20:57:20.343796 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 7df3aafea18f49249cceb8a9431064e6 (table default.Table_2d69d3afb9ca476cb5b329145ef8f74c [id=7016ed2e98194e4c89d508e5f724cf04]). Expected version 1 got 0
I20250110 20:57:20.339792 5155 tablet_service.cc:1467] Processing CreateTablet for tablet c010af5c95c5421eb6bb7439f2164498 (DEFAULT_TABLE table=default.Table_d28fb676474445e5bc7377940f7a4c3a [id=9f2cc4681a9c447293b0218067fc3ecb]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:20.349361 6121 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 0.679s user 0.200s sys 0.207s
W20250110 20:57:20.349695 6121 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 0.679s user 0.202s sys 0.207s
I20250110 20:57:20.358013 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_50279f2aee724e4792bbc5ccba5d0560"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:20.359546 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50279f2aee724e4792bbc5ccba5d0560 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:20.367985 4893 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet e324e1bdd5754864a0f65c50594c2d93
I20250110 20:57:20.375059 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 0e9c229f569c4468bf43e7e880f7f0df
I20250110 20:57:20.382683 5600 tablet_bootstrap.cc:492] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:20.385612 5303 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9c229f569c4468bf43e7e880f7f0df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:20.390429 5594 raft_consensus.cc:357] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.391258 5594 raft_consensus.cc:383] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:20.391547 5594 raft_consensus.cc:738] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
W20250110 20:57:20.329550 5462 thread.cc:635] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.010s sys 0.001s
W20250110 20:57:20.393644 5462 thread.cc:601] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s user 0.010s sys 0.001s
I20250110 20:57:20.393296 5594 consensus_queue.cc:260] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.395749 5594 ts_tablet_manager.cc:1428] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.472s user 0.005s sys 0.000s
I20250110 20:57:20.396560 5594 tablet_bootstrap.cc:492] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:20.398105 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet e324e1bdd5754864a0f65c50594c2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:20.398829 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet e324e1bdd5754864a0f65c50594c2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:20.399132 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9c229f569c4468bf43e7e880f7f0df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:20.405167 5460 raft_consensus.cc:2463] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:20.411643 5600 tablet_bootstrap.cc:654] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:20.423344 6387 tablet.cc:1721] T 2d5c46de95bc4227bffc5e8425b8cc95 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:20.434305 5773 ts_tablet_manager.cc:1918] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:20.434667 5773 log.cc:1198] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/cd0a666892d04e97a4b9967e38ab0946
I20250110 20:57:20.435312 5773 ts_tablet_manager.cc:1939] T cd0a666892d04e97a4b9967e38ab0946 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:20.439540 5597 tablet_bootstrap.cc:654] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:20.441437 5145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c27a964962af44c48f7fcfa9a130a08f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.443154 5155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c010af5c95c5421eb6bb7439f2164498. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.447310 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 319c544143824575bd96ae9d808fbb9c (DEFAULT_TABLE table=default.Table_d67d43097b044b11bff93e2e95a7922a [id=15c3fc4ad0c140d399d1ef10a8f7d3e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.448185 5153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 319c544143824575bd96ae9d808fbb9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.449201 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 948981b34bdd4017b7ce659e7f850c5f (DEFAULT_TABLE table=default.Table_f88b13f73f784b8890702e20bc102be2 [id=f99b7dcae8bb4a70b33f401a1cd1872e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.449775 5156 tablet_service.cc:1467] Processing CreateTablet for tablet 2591beedb8d2456da1c282d667b156a9 (DEFAULT_TABLE table=default.Table_652379995947455b9a395736e35ca034 [id=01dddc8bb7fd43fabe5fa3860a152cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.450048 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 948981b34bdd4017b7ce659e7f850c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.450614 5156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2591beedb8d2456da1c282d667b156a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.458817 5600 tablet_bootstrap.cc:492] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:20.459242 5600 ts_tablet_manager.cc:1397] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.004s
I20250110 20:57:20.467592 5595 ts_tablet_manager.cc:1918] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:20.467939 5595 log.cc:1198] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/bae52ea080f54d228b6b76922248ed5a
I20250110 20:57:20.468483 5595 ts_tablet_manager.cc:1939] T bae52ea080f54d228b6b76922248ed5a P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:20.461273 5600 raft_consensus.cc:357] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.478022 5600 raft_consensus.cc:383] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:20.478315 5600 raft_consensus.cc:738] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:20.478946 5600 consensus_queue.cc:260] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.481220 5600 ts_tablet_manager.cc:1428] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20250110 20:57:20.485828 5368 heartbeater.cc:346] Connected to a master server at 127.4.171.252:37233
I20250110 20:57:20.488266 6175 raft_consensus.cc:491] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.498127 5600 tablet_bootstrap.cc:492] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:20.497723 6175 raft_consensus.cc:513] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.499516 6175 leader_election.cc:290] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:20.505774 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 257) took 1116 ms. Trace:
I20250110 20:57:20.506068 5595 rpcz_store.cc:276] 0110 20:57:19.389325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.425871 (+ 36546us) service_pool.cc:225] Handling call
0110 20:57:20.296146 (+870275us) ts_tablet_manager.cc:1145] Deleting tablet bae52ea080f54d228b6b76922248ed5a
0110 20:57:20.296189 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:20.424847 (+128658us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.467390 (+ 42543us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.505707 (+ 38317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":869731}
I20250110 20:57:20.507184 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet bae52ea080f54d228b6b76922248ed5a (table default.Table_41125035ac2544fb9a27c3c25133b711 [id=4bf0e84fcbfb4b2a841abc862506a8c2]) successfully deleted
I20250110 20:57:20.508024 5595 tablet_replica.cc:331] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:20.508706 5595 raft_consensus.cc:2238] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:20.509156 5595 raft_consensus.cc:2267] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:20.455736 5141 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 260) took 1386 ms. Trace:
I20250110 20:57:20.456537 6269 raft_consensus.cc:2798] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:20.461987 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c813130649cc4df1b483ac4a86b4f0a0" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:20.479748 5151 tablet_service.cc:1467] Processing CreateTablet for tablet a97def2f16e94061b1f938f73465fd31 (DEFAULT_TABLE table=default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.472045 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 234) took 2452 ms. Trace:
I20250110 20:57:20.535871 5773 rpcz_store.cc:276] 0110 20:57:18.018772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.031418 (+ 12646us) service_pool.cc:225] Handling call
0110 20:57:19.788705 (+1757287us) ts_tablet_manager.cc:1145] Deleting tablet cd0a666892d04e97a4b9967e38ab0946
0110 20:57:19.788744 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:20.354623 (+565879us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.434092 (+ 79469us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.470786 (+ 36694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1756792}
I20250110 20:57:20.480644 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 210b8197624547d38c084def211b3244 (DEFAULT_TABLE table=default.Table_f5aec17b06764886ad9b70bf235a2da0 [id=37e03fb5b72a4d97b2564cb8f5dd8fee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.536993 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet cd0a666892d04e97a4b9967e38ab0946 (table default.Table_c3b44e7a68ac47a1ad0f7c5496130418 [id=143a11fac7564c92b6a7224e6ac31bfb]) successfully deleted
I20250110 20:57:20.490352 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 74fd501dc6284a2c9b1da666ee81f688 (DEFAULT_TABLE table=default.Table_2dfaee1ffeab4425896fad5e23dbb691 [id=36abbb2daa6648888440d2e9a579fef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.537866 5773 tablet_replica.cc:331] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
W20250110 20:57:20.512306 5143 consensus_meta.cc:287] T 4d9672dc9cbb4e27bf6b01f513b6ccb8 P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 0.845s user 0.007s sys 0.000s
I20250110 20:57:20.442998 6369 raft_consensus.cc:491] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.538543 5773 raft_consensus.cc:2238] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:20.539003 5773 raft_consensus.cc:2267] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:20.541163 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74fd501dc6284a2c9b1da666ee81f688. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.542675 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a97def2f16e94061b1f938f73465fd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.543164 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 210b8197624547d38c084def211b3244. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.545972 5595 ts_tablet_manager.cc:1905] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:20.549544 5600 tablet_bootstrap.cc:654] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:20.555400 6386 tablet.cc:1721] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:20.560245 5594 tablet_bootstrap.cc:654] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:20.569763 6355 consensus_meta.cc:287] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 0.910s user 0.007s sys 0.000s
I20250110 20:57:20.582576 6175 raft_consensus.cc:491] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.583001 6175 raft_consensus.cc:513] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.584818 6175 leader_election.cc:290] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:20.585598 5155 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e9c229f569c4468bf43e7e880f7f0df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:20.570550 6145 raft_consensus.cc:491] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46005a5920cc4a3697a5c61b1c271384)
I20250110 20:57:20.588667 6145 raft_consensus.cc:513] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:20.590400 6145 leader_election.cc:290] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:20.582103 6355 raft_consensus.cc:513] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:20.595005 5600 tablet_bootstrap.cc:492] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:20.595425 5600 ts_tablet_manager.cc:1397] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.097s user 0.010s sys 0.000s
I20250110 20:57:20.595904 5458 raft_consensus.cc:2388] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46005a5920cc4a3697a5c61b1c271384 in current term 1: Already voted for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in this term.
I20250110 20:57:20.598031 5600 raft_consensus.cc:357] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.598729 5600 raft_consensus.cc:383] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:20.599013 5600 raft_consensus.cc:738] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:20.599274 6355 leader_election.cc:290] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:20.599761 5600 consensus_queue.cc:260] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.606949 5600 ts_tablet_manager.cc:1428] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
W20250110 20:57:20.622747 5148 consensus_meta.cc:287] T a2bcc237beae4d90aee701b0ee1ae14f P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 0.582s user 0.006s sys 0.001s
I20250110 20:57:20.630820 5143 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 258) took 1576 ms. Trace:
I20250110 20:57:20.631287 5143 rpcz_store.cc:276] 0110 20:57:19.054809 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.055003 (+ 194us) service_pool.cc:225] Handling call
0110 20:57:19.453438 (+398435us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.453574 (+ 136us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.666767 (+213193us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.630360 (+963593us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.630765 (+ 405us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:20.642300 5600 tablet_bootstrap.cc:492] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:20.649637 5773 ts_tablet_manager.cc:1905] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:20.659724 5594 tablet_bootstrap.cc:492] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:20.660207 4894 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_ae86414449b14c9aba13ee321887871a" } new_table_name: "default.Table_cdd71d3533f447558e49e1c6d6765f82" modify_external_catalogs: true
I20250110 20:57:20.660552 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_06885b5a5dd84257ac0f7be72678888f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:20.661440 4899 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5f0518ca447a401eb8fe8655ecdc9a55" } modify_external_catalogs: true
W20250110 20:57:20.661906 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06885b5a5dd84257ac0f7be72678888f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:20.662580 5594 ts_tablet_manager.cc:1397] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.266s user 0.002s sys 0.007s
I20250110 20:57:20.664721 5594 raft_consensus.cc:357] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.665349 5594 raft_consensus.cc:383] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:20.665280 4900 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_68999c6dd9a84b24b631302d58de6c02" } modify_external_catalogs: true
I20250110 20:57:20.665632 5594 raft_consensus.cc:738] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:20.665757 4900 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_68999c6dd9a84b24b631302d58de6c02" } modify_external_catalogs: true
I20250110 20:57:20.666020 4893 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_3cccb8b0fa3f403391692f5f3c691637" } new_table_name: "default.Table_853611a0967d473eab31c83128ebe9fc" modify_external_catalogs: true
I20250110 20:57:20.666146 5594 consensus_queue.cc:260] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.661916 4899 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_5f0518ca447a401eb8fe8655ecdc9a55" } modify_external_catalogs: true
I20250110 20:57:20.641007 5595 ts_tablet_manager.cc:1918] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:20.668435 5595 log.cc:1198] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/da6a9a054aa54455b4979e4c633ad6b4
I20250110 20:57:20.669179 5595 ts_tablet_manager.cc:1939] T da6a9a054aa54455b4979e4c633ad6b4 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:20.669459 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_b2e57cd89b604d4b80eadaf846bba725"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:20.670840 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2e57cd89b604d4b80eadaf846bba725 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:20.675297 5600 tablet_bootstrap.cc:654] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:20.669384 5592 ts_tablet_manager.cc:1918] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:20.680991 5592 log.cc:1198] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/7462d21b91c2441fb9e11383466e9ca7
I20250110 20:57:20.681733 5592 ts_tablet_manager.cc:1939] T 7462d21b91c2441fb9e11383466e9ca7 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:20.683579 5594 ts_tablet_manager.cc:1428] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250110 20:57:20.680331 5595 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55028 (request call id 259) took 1254 ms. Trace:
I20250110 20:57:20.697535 5595 rpcz_store.cc:276] 0110 20:57:19.426015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.428658 (+ 2643us) service_pool.cc:225] Handling call
0110 20:57:20.507886 (+1079228us) ts_tablet_manager.cc:1145] Deleting tablet da6a9a054aa54455b4979e4c633ad6b4
0110 20:57:20.507927 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:20.576953 (+ 69026us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.640813 (+ 63860us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.680275 (+ 39462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078552}
I20250110 20:57:20.699126 5595 tablet_replica.cc:331] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:20.699797 5595 raft_consensus.cc:2238] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:20.700194 5595 raft_consensus.cc:2267] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:20.702169 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_3cccb8b0fa3f403391692f5f3c691637 [id=b65f8d62ac7f44169142988e1c4903d9]) successfully deleted
W20250110 20:57:20.702780 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet dd07a9eaa4364d1ca5b577a010978899 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd07a9eaa4364d1ca5b577a010978899 already in progress: creating tablet
I20250110 20:57:20.702924 5773 ts_tablet_manager.cc:1918] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:20.703243 5773 log.cc:1198] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/fde470279464496aa7efdee3800754f3
I20250110 20:57:20.703725 5595 ts_tablet_manager.cc:1905] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:20.703919 5773 ts_tablet_manager.cc:1939] T fde470279464496aa7efdee3800754f3 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
W20250110 20:57:20.636363 5170 thread.cc:635] prepare [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.022s sys 0.025s
I20250110 20:57:20.584785 5149 tablet_service.cc:1514] Processing DeleteTablet for tablet e324e1bdd5754864a0f65c50594c2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:20.704517 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 367110us
I20250110 20:57:20.624182 6269 raft_consensus.cc:695] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
W20250110 20:57:20.730732 5111 thread.cc:635] raft [worker] (thread pool) Time spent creating pthread: real 1.055s user 0.004s sys 0.006s
W20250110 20:57:20.731060 5111 thread.cc:601] raft [worker] (thread pool) Time spent starting thread: real 1.056s user 0.004s sys 0.006s
W20250110 20:57:20.720815 5170 thread.cc:601] prepare [worker] (thread pool) Time spent starting thread: real 0.992s user 0.022s sys 0.025s
I20250110 20:57:20.740679 5600 tablet_bootstrap.cc:492] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:20.741453 5600 ts_tablet_manager.cc:1397] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.099s user 0.007s sys 0.002s
I20250110 20:57:20.744055 5600 raft_consensus.cc:357] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:20.744783 5600 raft_consensus.cc:383] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:20.745070 5600 raft_consensus.cc:738] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
W20250110 20:57:20.742904 5138 consensus_meta.cc:287] T 4d0cee6de1f94451ba3530f684226055 P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 0.569s user 0.001s sys 0.004s
W20250110 20:57:20.743471 5146 consensus_meta.cc:287] T d2b789aad72643149da01b9a607e2ce5 P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 1.115s user 0.007s sys 0.000s
W20250110 20:57:20.744139 5152 consensus_meta.cc:287] T 97dfd451501d49909b3d163a53548ce4 P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 1.073s user 0.007s sys 0.000s
I20250110 20:57:20.753084 5595 ts_tablet_manager.cc:1918] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 20:57:20.779080 5142 consensus_meta.cc:287] T d26c2bd25161489baa156e78128e35aa P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 0.812s user 0.005s sys 0.000s
I20250110 20:57:20.781592 5595 log.cc:1198] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/c813130649cc4df1b483ac4a86b4f0a0
I20250110 20:57:20.782336 5595 ts_tablet_manager.cc:1939] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:20.767552 5600 consensus_queue.cc:260] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:20.784256 5600 ts_tablet_manager.cc:1428] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.042s user 0.004s sys 0.000s
I20250110 20:57:20.779798 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758430b311a8478c9cab2bcf79253b52" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:20.584673 6369 raft_consensus.cc:513] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.790659 6369 leader_election.cc:290] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:20.785794 6269 consensus_queue.cc:237] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.536370 5141 rpcz_store.cc:276] 0110 20:57:19.069168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.069344 (+ 176us) service_pool.cc:225] Handling call
0110 20:57:19.560945 (+491601us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.587424 (+ 26479us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:20.024221 (+436797us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.454779 (+430558us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.455683 (+ 904us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:20.798233 6311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84772eba3f0460faf8523f76eb57e53. Latest consensus state: current_term: 2 leader_uuid: "b84772eba3f0460faf8523f76eb57e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:20.798951 6311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:20.799858 5600 tablet_bootstrap.cc:492] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:20.802551 6348 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b84772eba3f0460faf8523f76eb57e53)
I20250110 20:57:20.803092 6348 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:20.805810 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet c813130649cc4df1b483ac4a86b4f0a0 (table default.Table_e17e3d34a10d4c05b4ae644cd35c9263 [id=99a3fb146486431188b7bc452ea806ad]) successfully deleted
W20250110 20:57:20.809952 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 5feb54cd239b4a80894d45898776fce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5feb54cd239b4a80894d45898776fce5 already in progress: creating tablet
I20250110 20:57:20.809779 6368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b84772eba3f0460faf8523f76eb57e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:20.810545 6368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:20.807951 6348 leader_election.cc:290] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381)
W20250110 20:57:20.816641 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 0e9c229f569c4468bf43e7e880f7f0df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e9c229f569c4468bf43e7e880f7f0df already in progress: creating tablet
I20250110 20:57:20.826606 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:20.820734 5147 consensus_meta.cc:287] T 36bca826ec144f77b96b4a41277fd6d5 P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 1.102s user 0.002s sys 0.007s
I20250110 20:57:20.833782 6121 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44808 (request call id 0) took 4317 ms. Trace:
I20250110 20:57:20.835217 5152 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 249) took 1840 ms. Trace:
I20250110 20:57:20.835642 5152 rpcz_store.cc:276] 0110 20:57:18.994437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.994555 (+ 118us) service_pool.cc:225] Handling call
0110 20:57:19.413131 (+418576us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.414608 (+ 1477us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.670789 (+256181us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.834807 (+1164018us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.835162 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":68247}
I20250110 20:57:20.805266 5600 tablet_bootstrap.cc:654] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:20.844725 5139 consensus_meta.cc:287] T 55542f8735d141789205e602651d38c6 P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 1.064s user 0.006s sys 0.003s
I20250110 20:57:20.861778 5600 tablet_bootstrap.cc:492] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:20.862185 5600 ts_tablet_manager.cc:1397] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.000s
I20250110 20:57:20.864482 5600 raft_consensus.cc:357] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.864985 6401 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250110 20:57:20.865164 5600 raft_consensus.cc:383] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:20.871737 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet e324e1bdd5754864a0f65c50594c2d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e324e1bdd5754864a0f65c50594c2d93 already in progress: creating tablet
W20250110 20:57:20.865978 5140 consensus_meta.cc:287] T 873b9a1c522d4f7895740db18a537a1b P ac9003be04f340f3886b5625cc928eb0: Time spent flushing consensus metadata: real 0.711s user 0.004s sys 0.004s
I20250110 20:57:20.866843 5138 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 263) took 1768 ms. Trace:
I20250110 20:57:20.868427 5173 raft_consensus.cc:3054] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:20.870054 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e53d95ef8645c0bd123138655305ac" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:20.871203 5175 raft_consensus.cc:3054] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:20.879524 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:20.886065 5600 raft_consensus.cc:738] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:20.886893 5600 consensus_queue.cc:260] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.893543 5138 rpcz_store.cc:276] 0110 20:57:19.098024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.098208 (+ 184us) service_pool.cc:225] Handling call
0110 20:57:19.760989 (+662781us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.761091 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:20.159293 (+398202us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.866338 (+707045us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.866785 (+ 447us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:20.895485 6059 raft_consensus.cc:491] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.895941 6059 raft_consensus.cc:513] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:20.897982 6059 leader_election.cc:290] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:20.899363 5600 ts_tablet_manager.cc:1428] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.037s user 0.005s sys 0.000s
I20250110 20:57:20.901751 5597 tablet_bootstrap.cc:492] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:20.909461 5321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:20.918584 5600 tablet_bootstrap.cc:492] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:20.902894 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bda90a78a5b4e2a839d496b9317f6c0" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:20.927510 5597 ts_tablet_manager.cc:1397] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.964s user 0.017s sys 0.003s
I20250110 20:57:20.903848 5140 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 261) took 1821 ms. Trace:
I20250110 20:57:20.932439 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:57:20.935650 5600 tablet_bootstrap.cc:654] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:20.949633 6104 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" instance_seqno: 1736542619288207) as {username='slave'} at 127.4.171.194:34013; Asking this server to re-register.
W20250110 20:57:20.959896 5257 leader_election.cc:343] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:20.960325 5257 leader_election.cc:304] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:20.960837 5257 raft_consensus.cc:2681] T c813130649cc4df1b483ac4a86b4f0a0 P 46005a5920cc4a3697a5c61b1c271384: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250110 20:57:20.961923 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 7df3aafea18f49249cceb8a9431064e6 (table default.Table_2d69d3afb9ca476cb5b329145ef8f74c [id=7016ed2e98194e4c89d508e5f724cf04]). Expected version 1 got 0
W20250110 20:57:20.962327 5257 leader_election.cc:343] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:20.962661 5257 leader_election.cc:304] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:20.964553 6175 raft_consensus.cc:2743] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:20.964903 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 1926f6b908cd4053a52b08ca22580ccf (DEFAULT_TABLE table=default.Table_f982d5a7aa86433da097e579045584df [id=d8a74e8af23147ec9e966973dd1009af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.963833 5293 tablet_service.cc:1467] Processing CreateTablet for tablet a67db53acbe84ce6848d6156245f2bf8 (DEFAULT_TABLE table=default.Table_6529554ce9fa48d482b0343488159c52 [id=cfa07e022c454aada9c5793b42d5e917]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:20.965698 5257 leader_election.cc:343] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:20.966349 5257 leader_election.cc:343] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:20.968454 5289 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0b523c83df421a8db74c0e5467f988 (DEFAULT_TABLE table=default.Table_fe05cd0420bb4aaa86a1e3789596e738 [id=8920ffdc30674ca6bade3cafe3ca76e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.970623 5368 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:20.971909 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 5c038a7c42ea429b886f6dad03dca510 (DEFAULT_TABLE table=default.Table_6200af64376b455fac5a660790cd0e90 [id=c057edfdb4c64b82aee98254ce29ee92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:20.974809 5290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1926f6b908cd4053a52b08ca22580ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.982858 5289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e0b523c83df421a8db74c0e5467f988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.983400 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c038a7c42ea429b886f6dad03dca510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.985512 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a67db53acbe84ce6848d6156245f2bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:20.988251 6175 raft_consensus.cc:491] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:20.988648 6175 raft_consensus.cc:513] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:20.990374 6175 leader_election.cc:290] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:21.001291 5110 leader_election.cc:304] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters:
I20250110 20:57:21.004909 6175 raft_consensus.cc:491] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.007139 6175 raft_consensus.cc:513] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.005954 5255 leader_election.cc:304] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:20.912652 5111 consensus_peers.cc:487] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c5b954dbccb45b2a6ab7cc5cceadffb. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:21.010906 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 98330.9us
I20250110 20:57:21.015445 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 5c038a7c42ea429b886f6dad03dca510 (DEFAULT_TABLE table=default.Table_6200af64376b455fac5a660790cd0e90 [id=c057edfdb4c64b82aee98254ce29ee92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.017623 6175 leader_election.cc:290] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:21.018844 6175 raft_consensus.cc:3054] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.022586 5427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c038a7c42ea429b886f6dad03dca510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.029810 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_559c738b24f74dd78374a3dbc5f1dfab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:21.031265 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_559c738b24f74dd78374a3dbc5f1dfab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:21.032260 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_e1b3189527784d23b921008fd95b28f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:21.036633 5426 consensus_meta.cc:287] T dd07a9eaa4364d1ca5b577a010978899 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 0.629s user 0.004s sys 0.006s
I20250110 20:57:21.037323 4896 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:21.039882 6107 ts_manager.cc:194] Registered new tserver with Master: 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:21.040210 5430 consensus_meta.cc:287] T 700374c672fe43c9a45f34d0255b8683 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 0.621s user 0.002s sys 0.004s
W20250110 20:57:21.043272 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1b3189527784d23b921008fd95b28f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:21.045385 5428 consensus_meta.cc:287] T 2591beedb8d2456da1c282d667b156a9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 0.629s user 0.010s sys 0.000s
I20250110 20:57:21.050951 5141 tablet_service.cc:1467] Processing CreateTablet for tablet 1926f6b908cd4053a52b08ca22580ccf (DEFAULT_TABLE table=default.Table_f982d5a7aa86433da097e579045584df [id=d8a74e8af23147ec9e966973dd1009af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.051844 5137 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 264) took 1937 ms. Trace:
I20250110 20:57:21.052181 5137 rpcz_store.cc:276] 0110 20:57:19.114553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.117540 (+ 2987us) service_pool.cc:225] Handling call
0110 20:57:19.765455 (+647915us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.765546 (+ 91us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:20.151676 (+386130us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.725799 (+574123us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.051786 (+325987us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":157088}
W20250110 20:57:21.052942 5432 consensus_meta.cc:287] T 6b8f59800b2a49a1b69b43133d4d0551 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 1.192s user 0.005s sys 0.001s
I20250110 20:57:21.055452 6390 raft_consensus.cc:2798] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:21.055752 6390 raft_consensus.cc:491] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:21.056036 6390 raft_consensus.cc:3054] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.057683 5600 tablet_bootstrap.cc:492] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:21.058061 5600 ts_tablet_manager.cc:1397] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.140s user 0.010s sys 0.000s
I20250110 20:57:21.060084 5600 raft_consensus.cc:357] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.060717 5600 raft_consensus.cc:383] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.060952 5600 raft_consensus.cc:738] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.062455 6104 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" instance_seqno: 1736542621014757) as {username='slave'} at 127.4.171.195:46805; Asking this server to re-register.
I20250110 20:57:21.067533 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f80685cb3c494283bea421195bf7751f (table default.Table_3f641bf1cdfd46fda1bd487fa4f01f1e [id=75dcee4b73484274bac56a2ab1a827d7]). Expected version 1 got 0
I20250110 20:57:21.067793 6175 raft_consensus.cc:2743] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:21.070693 5433 consensus_meta.cc:287] T a2bcc237beae4d90aee701b0ee1ae14f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 0.648s user 0.002s sys 0.004s
I20250110 20:57:21.072127 5435 tablet_service.cc:1467] Processing CreateTablet for tablet 1926f6b908cd4053a52b08ca22580ccf (DEFAULT_TABLE table=default.Table_f982d5a7aa86433da097e579045584df [id=d8a74e8af23147ec9e966973dd1009af]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.072803 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e53d95ef8645c0bd123138655305ac with delete_type TABLET_DATA_DELETED (replaced by tablet ba0d83a50ec0479f9673a57370b0e879) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:21.073611 5435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1926f6b908cd4053a52b08ca22580ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.076860 5430 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40232 (request call id 284) took 1243 ms. Trace:
I20250110 20:57:21.077224 5430 rpcz_store.cc:276] 0110 20:57:19.845878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.847824 (+ 1946us) service_pool.cc:225] Handling call
0110 20:57:19.848984 (+ 1160us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.849055 (+ 71us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:20.419012 (+569957us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.076383 (+657371us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.076795 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35729}
I20250110 20:57:21.079598 5426 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40232 (request call id 279) took 1265 ms. Trace:
I20250110 20:57:21.079916 5426 rpcz_store.cc:276] 0110 20:57:19.814263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.821694 (+ 7431us) service_pool.cc:225] Handling call
0110 20:57:19.822822 (+ 1128us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.822890 (+ 68us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:20.407541 (+584651us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.079135 (+671594us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.079550 (+ 415us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41886}
I20250110 20:57:21.081923 5431 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40232 (request call id 278) took 1267 ms. Trace:
I20250110 20:57:21.082224 5431 rpcz_store.cc:276] 0110 20:57:19.813971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.814684 (+ 713us) service_pool.cc:225] Handling call
0110 20:57:19.815892 (+ 1208us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.815967 (+ 75us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:21.041689 (+1225722us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.081428 (+ 39739us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.081790 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33668}
I20250110 20:57:21.084106 5432 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40232 (request call id 281) took 1251 ms. Trace:
I20250110 20:57:21.084494 5600 consensus_queue.cc:260] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.084481 5432 rpcz_store.cc:276] 0110 20:57:19.832646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.833490 (+ 844us) service_pool.cc:225] Handling call
0110 20:57:19.834752 (+ 1262us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.834824 (+ 72us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.860022 (+ 25198us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.083695 (+1223673us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.084060 (+ 365us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":30286}
W20250110 20:57:21.087927 5398 leader_election.cc:343] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: fde470279464496aa7efdee3800754f3
I20250110 20:57:21.088804 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:21.090320 5433 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40232 (request call id 282) took 1257 ms. Trace:
I20250110 20:57:21.090670 5433 rpcz_store.cc:276] 0110 20:57:19.832953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.837486 (+ 4533us) service_pool.cc:225] Handling call
0110 20:57:19.838643 (+ 1157us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.838717 (+ 74us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:20.421977 (+583260us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.089881 (+667904us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.090271 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14983}
I20250110 20:57:21.096046 5600 ts_tablet_manager.cc:1428] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.038s user 0.004s sys 0.001s
I20250110 20:57:21.101950 5429 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40232 (request call id 280) took 1287 ms. Trace:
I20250110 20:57:21.102473 5429 rpcz_store.cc:276] 0110 20:57:19.814559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.817543 (+ 2984us) service_pool.cc:225] Handling call
0110 20:57:19.818667 (+ 1124us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.818739 (+ 72us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.909234 (+ 90495us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.101390 (+1192156us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.101816 (+ 426us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":70655}
I20250110 20:57:21.104061 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80685cb3c494283bea421195bf7751f" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.106477 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 232) took 3135 ms. Trace:
I20250110 20:57:21.107335 5147 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 250) took 2103 ms. Trace:
I20250110 20:57:21.107654 5147 rpcz_store.cc:276] 0110 20:57:19.004279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.004415 (+ 136us) service_pool.cc:225] Handling call
0110 20:57:19.417465 (+413050us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.417567 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.712243 (+294676us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.053181 (+1340938us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.107276 (+ 54095us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":157732}
I20250110 20:57:21.109668 5146 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 252) took 2095 ms. Trace:
I20250110 20:57:21.110000 5146 rpcz_store.cc:276] 0110 20:57:19.014361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.014487 (+ 126us) service_pool.cc:225] Handling call
0110 20:57:19.422216 (+407729us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.422316 (+ 100us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.627776 (+205460us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.834496 (+1206720us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.109366 (+274870us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c98c2 00007f896525b999 00007f89644c05f8 00007f89645840c3 00007f896457fe8f 00007f896458064d 00007f8964531859 00007f8962c19c64 00007f8962c19bf2
0110 20:57:21.109618 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":222835,"spinlock_wait_cycles":10450176}
I20250110 20:57:21.115542 5773 rpcz_store.cc:276] 0110 20:57:17.970855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.999614 (+ 28759us) service_pool.cc:225] Handling call
0110 20:57:20.537737 (+2538123us) ts_tablet_manager.cc:1145] Deleting tablet fde470279464496aa7efdee3800754f3
0110 20:57:20.537775 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:20.656001 (+118226us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.702686 (+ 46685us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.106425 (+403739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2503911}
I20250110 20:57:21.116393 5600 tablet_bootstrap.cc:492] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:21.116631 5773 tablet_replica.cc:331] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:21.117291 5773 raft_consensus.cc:2238] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:21.118388 5142 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 259) took 2053 ms. Trace:
I20250110 20:57:21.118685 5142 rpcz_store.cc:276] 0110 20:57:19.065161 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.065340 (+ 179us) service_pool.cc:225] Handling call
0110 20:57:19.456010 (+390670us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.509480 (+ 53470us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.705077 (+195597us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.050079 (+1345002us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.118107 (+ 68028us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c98c2 00007f896525b999 00007f89644c05f8 00007f89645840c3 00007f896457fe8f 00007f896458064d 00007f8964531859 00007f8962c19c64 00007f8962c19bf2
0110 20:57:21.118337 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":182316,"spinlock_wait_cycles":10134272}
I20250110 20:57:21.120111 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 5c038a7c42ea429b886f6dad03dca510 (DEFAULT_TABLE table=default.Table_6200af64376b455fac5a660790cd0e90 [id=c057edfdb4c64b82aee98254ce29ee92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.121089 5138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c038a7c42ea429b886f6dad03dca510. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.126649 5139 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35166 (request call id 262) took 2037 ms. Trace:
I20250110 20:57:21.126961 5139 rpcz_store.cc:276] 0110 20:57:19.089659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.089841 (+ 182us) service_pool.cc:225] Handling call
0110 20:57:19.755790 (+665949us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.755889 (+ 99us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.780718 (+ 24829us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.985787 (+1205069us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.126598 (+140811us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:21.128944 5141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1926f6b908cd4053a52b08ca22580ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.132412 5319 raft_consensus.cc:2463] T 371fb7408f0149c8b00659e09ebaf48a P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:21.135778 5600 tablet_bootstrap.cc:654] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:21.144821 5257 leader_election.cc:343] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:21.150565 5110 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:21.110369 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0b523c83df421a8db74c0e5467f988 (DEFAULT_TABLE table=default.Table_fe05cd0420bb4aaa86a1e3789596e738 [id=8920ffdc30674ca6bade3cafe3ca76e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.111871 5444 tablet_service.cc:1467] Processing CreateTablet for tablet a67db53acbe84ce6848d6156245f2bf8 (DEFAULT_TABLE table=default.Table_6529554ce9fa48d482b0343488159c52 [id=cfa07e022c454aada9c5793b42d5e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.114478 5460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af278622ef6432fb0891cb06bc6cd6d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.116782 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.135764 5440 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40232 (request call id 283) took 1302 ms. Trace:
W20250110 20:57:21.140349 5441 consensus_meta.cc:287] T c010af5c95c5421eb6bb7439f2164498 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent flushing consensus metadata: real 0.767s user 0.006s sys 0.000s
I20250110 20:57:21.146765 5594 tablet_bootstrap.cc:492] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:21.147330 6356 raft_consensus.cc:491] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.148849 6205 raft_consensus.cc:491] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46005a5920cc4a3697a5c61b1c271384)
I20250110 20:57:21.151845 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 204) took 5220 ms. Trace:
I20250110 20:57:21.157326 5592 rpcz_store.cc:276] 0110 20:57:15.931799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.934342 (+ 2543us) service_pool.cc:225] Handling call
0110 20:57:20.271227 (+4336885us) ts_tablet_manager.cc:1145] Deleting tablet 7462d21b91c2441fb9e11383466e9ca7
0110 20:57:20.271265 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:20.602617 (+331352us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.669181 (+ 66564us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.151669 (+482488us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":513143,"spinlock_wait_cycles":2944,"tablet-delete.queue_time_us":4335766}
I20250110 20:57:21.166096 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e0b523c83df421a8db74c0e5467f988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.157135 5149 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0b523c83df421a8db74c0e5467f988 (DEFAULT_TABLE table=default.Table_fe05cd0420bb4aaa86a1e3789596e738 [id=8920ffdc30674ca6bade3cafe3ca76e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.168181 6205 raft_consensus.cc:513] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:20.944978 5140 rpcz_store.cc:276] 0110 20:57:19.082361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.082535 (+ 174us) service_pool.cc:225] Handling call
0110 20:57:19.751358 (+668823us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.751455 (+ 97us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:20.154920 (+403465us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:20.903403 (+748483us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:20.903798 (+ 395us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:21.167207 5152 tablet_service.cc:1467] Processing CreateTablet for tablet a67db53acbe84ce6848d6156245f2bf8 (DEFAULT_TABLE table=default.Table_6529554ce9fa48d482b0343488159c52 [id=cfa07e022c454aada9c5793b42d5e917]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:21.168632 6356 raft_consensus.cc:513] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.171595 6205 leader_election.cc:290] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:21.172479 6356 leader_election.cc:290] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.173000 5463 raft_consensus.cc:2463] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:21.175670 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a67db53acbe84ce6848d6156245f2bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.156832 5440 rpcz_store.cc:276] 0110 20:57:19.833222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.841493 (+ 8271us) service_pool.cc:225] Handling call
0110 20:57:19.842675 (+ 1182us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:19.842748 (+ 73us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:19.948349 (+105601us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.135120 (+1186771us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:21.135616 (+ 496us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":539454}
I20250110 20:57:21.183858 6175 raft_consensus.cc:491] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.184276 6175 raft_consensus.cc:513] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.186199 6175 leader_election.cc:290] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:21.187098 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 203) took 5281 ms. Trace:
I20250110 20:57:21.187523 5592 rpcz_store.cc:276] 0110 20:57:15.906108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.935645 (+ 29537us) service_pool.cc:225] Handling call
0110 20:57:21.186598 (+5250953us) ts_tablet_manager.cc:1145] Deleting tablet c57941aee12a4ebea2b47f4727fc6eee
0110 20:57:21.186702 (+ 104us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.187045 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5249083}
I20250110 20:57:21.188800 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 205) took 5247 ms. Trace:
I20250110 20:57:21.189236 5592 rpcz_store.cc:276] 0110 20:57:15.941637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:15.942840 (+ 1203us) service_pool.cc:225] Handling call
0110 20:57:21.188369 (+5245529us) ts_tablet_manager.cc:1145] Deleting tablet bae52ea080f54d228b6b76922248ed5a
0110 20:57:21.188428 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.188729 (+ 301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5204900}
I20250110 20:57:21.190996 5592 tablet_replica.cc:331] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:21.173702 5111 leader_election.cc:343] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 637a39e4264b40dd9480dc13e37359d9
W20250110 20:57:21.192155 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 180185us
I20250110 20:57:21.192438 5594 tablet_bootstrap.cc:654] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.194872 5773 pending_rounds.cc:62] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:21.195109 5773 pending_rounds.cc:69] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878646033891328 op_type: NO_OP noop_request { }
I20250110 20:57:21.195461 5773 raft_consensus.cc:2883] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:21.195771 5773 raft_consensus.cc:2267] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:21.207384 5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e0b523c83df421a8db74c0e5467f988. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.220306 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:57:21.222622 6034 leader_election.cc:304] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14d1e2d5c72f498f915c367e8750361f; no voters: 63aa77ee6ec84a6e82132b2de9e77486, b84772eba3f0460faf8523f76eb57e53
I20250110 20:57:21.223837 6348 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250110 20:57:21.224476 5111 leader_election.cc:343] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: fcc7b1acf4114bf69d801437855760df
W20250110 20:57:21.226027 5398 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 131144us
W20250110 20:57:21.225346 5111 leader_election.cc:343] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:21.226781 5111 leader_election.cc:304] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:21.231416 6417 raft_consensus.cc:491] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46005a5920cc4a3697a5c61b1c271384)
I20250110 20:57:21.231457 6390 raft_consensus.cc:513] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.231859 6417 raft_consensus.cc:513] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:21.233775 6417 leader_election.cc:290] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.237426 5594 tablet_bootstrap.cc:492] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:21.237803 5594 ts_tablet_manager.cc:1397] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.001s
I20250110 20:57:21.239763 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:21.240335 5319 raft_consensus.cc:2463] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:21.248204 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a67db53acbe84ce6848d6156245f2bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:21.256942 5398 leader_election.cc:304] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:21.262365 5597 raft_consensus.cc:357] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.263139 5597 raft_consensus.cc:383] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.263379 5597 raft_consensus.cc:738] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.263878 5597 consensus_queue.cc:260] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.265661 5174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df3aafea18f49249cceb8a9431064e6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.268275 6417 raft_consensus.cc:2798] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:21.268724 6417 raft_consensus.cc:695] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
W20250110 20:57:21.269356 5398 consensus_peers.cc:487] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da6a9a054aa54455b4979e4c633ad6b4. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:21.269924 5398 leader_election.cc:343] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:21.269622 6417 consensus_queue.cc:237] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:21.270490 6399 log.cc:926] Time spent T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0: Append to log took a long time: real 0.294s user 0.001s sys 0.000s
I20250110 20:57:21.270439 5398 leader_election.cc:304] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:21.271277 6356 raft_consensus.cc:2798] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:21.271631 6356 raft_consensus.cc:491] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:21.271989 6356 raft_consensus.cc:3054] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:21.274992 5398 leader_election.cc:343] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 01962c5b7ae64f80b998948d18250b4c
I20250110 20:57:21.277335 5594 raft_consensus.cc:357] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.277987 5594 raft_consensus.cc:383] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.278249 5594 raft_consensus.cc:738] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.278757 5594 consensus_queue.cc:260] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.280475 5594 ts_tablet_manager.cc:1428] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.042s user 0.003s sys 0.000s
W20250110 20:57:21.282147 5399 leader_election.cc:343] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.269553 6269 raft_consensus.cc:2743] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:21.283036 5399 consensus_peers.cc:487] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:21.283973 5399 leader_election.cc:343] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.284307 5399 leader_election.cc:304] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.285068 6415 raft_consensus.cc:2743] T 48e53d95ef8645c0bd123138655305ac P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:21.289033 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bda90a78a5b4e2a839d496b9317f6c0" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.233194 6390 leader_election.cc:290] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.278767 6356 raft_consensus.cc:513] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.307653 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:21.308168 5319 raft_consensus.cc:3054] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.309474 5594 tablet_bootstrap.cc:492] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:21.312886 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7df3aafea18f49249cceb8a9431064e6" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.314234 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80685cb3c494283bea421195bf7751f" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.314580 5319 raft_consensus.cc:2463] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:21.315680 5398 leader_election.cc:304] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:21.316469 6419 raft_consensus.cc:2798] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:21.316881 6419 raft_consensus.cc:695] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:21.317812 6419 consensus_queue.cc:237] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.321307 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a687769f69d844cc8564e61dc324d2b2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.322829 5255 leader_election.cc:304] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.323463 6175 raft_consensus.cc:2758] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250110 20:57:21.326951 6356 leader_election.cc:290] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.331171 5600 tablet_bootstrap.cc:492] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:21.331597 5600 ts_tablet_manager.cc:1397] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.215s user 0.031s sys 0.027s
W20250110 20:57:21.331979 5111 leader_election.cc:343] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: fcc7b1acf4114bf69d801437855760df
I20250110 20:57:21.333832 5600 raft_consensus.cc:357] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.334439 5600 raft_consensus.cc:383] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.334542 5592 raft_consensus.cc:2238] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:21.334663 5600 raft_consensus.cc:738] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.335180 5592 raft_consensus.cc:2267] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:21.336825 5592 ts_tablet_manager.cc:1905] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:21.342060 5600 consensus_queue.cc:260] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.344197 4895 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 33db8a2fdd9c4fde985345604fd04215 (table default.Table_094c089272f6475a989a44b5e1843fd5 [id=505b9ca1a68e401aa6f6f28a41f09181]). Expected version 1 got 0
I20250110 20:57:21.138361 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet fde470279464496aa7efdee3800754f3 (table default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]) successfully deleted
W20250110 20:57:21.352485 4887 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 214025us
W20250110 20:57:21.335631 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_5675215f702a41ab934b6baf97fd0431 [id=1c2b9e8ff5d341c380b50023c577d0aa], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7309d13f62524b6ab8b4c551728ecb6d consensus state has no leader
I20250110 20:57:21.339082 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 7462d21b91c2441fb9e11383466e9ca7 (table default.Table_447ad91cdb774363a84b85d71a154011 [id=12aaff501d444161b23ca0484538a854]) successfully deleted
W20250110 20:57:21.356581 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet c57941aee12a4ebea2b47f4727fc6eee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c57941aee12a4ebea2b47f4727fc6eee already in progress: creating tablet
W20250110 20:57:21.357427 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet bae52ea080f54d228b6b76922248ed5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bae52ea080f54d228b6b76922248ed5a already in progress: creating tablet
I20250110 20:57:21.359536 5600 ts_tablet_manager.cc:1428] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.028s user 0.000s sys 0.004s
I20250110 20:57:21.365083 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_ac4a9fe3dc414f02b5e85036973f1fd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:21.367203 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4a9fe3dc414f02b5e85036973f1fd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:21.383355 5600 tablet_bootstrap.cc:492] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:21.385210 5592 ts_tablet_manager.cc:1918] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:21.385740 5592 log.cc:1198] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/c44ca47c747544b6afec83b399dcdcb5
I20250110 20:57:21.386575 5592 ts_tablet_manager.cc:1939] T c44ca47c747544b6afec83b399dcdcb5 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:21.394774 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 207) took 4980 ms. Trace:
I20250110 20:57:21.396536 5600 tablet_bootstrap.cc:654] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.396631 5592 rpcz_store.cc:276] 0110 20:57:16.414026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.414502 (+ 476us) service_pool.cc:225] Handling call
0110 20:57:21.190844 (+4776342us) ts_tablet_manager.cc:1145] Deleting tablet c44ca47c747544b6afec83b399dcdcb5
0110 20:57:21.190883 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.344306 (+153423us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.378237 (+ 33931us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.394714 (+ 16477us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":142813,"tablet-delete.queue_time_us":4775806}
I20250110 20:57:21.399109 6175 raft_consensus.cc:491] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.399538 6175 raft_consensus.cc:513] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.400590 5592 tablet_replica.cc:331] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:21.401302 6175 leader_election.cc:290] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:21.402813 5592 raft_consensus.cc:2238] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:21.403273 5592 raft_consensus.cc:2267] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:21.407485 5592 ts_tablet_manager.cc:1905] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:21.413662 5255 leader_election.cc:343] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.414011 5255 leader_election.cc:304] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.414829 6175 raft_consensus.cc:2743] T 1bda90a78a5b4e2a839d496b9317f6c0 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:21.422712 5594 tablet_bootstrap.cc:654] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:21.425326 5255 leader_election.cc:343] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:21.426303 5255 leader_election.cc:343] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.426623 5255 leader_election.cc:304] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.427527 6175 raft_consensus.cc:2743] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:21.428992 5255 leader_election.cc:343] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.436249 6356 raft_consensus.cc:491] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.436447 6104 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" instance_seqno: 1736542617466493) as {username='slave'} at 127.4.171.193:50037; Asking this server to re-register.
I20250110 20:57:21.436791 6356 raft_consensus.cc:513] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.439677 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:21.440284 5319 raft_consensus.cc:2463] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:21.441027 5509 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:21.441321 5398 leader_election.cc:304] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:21.442130 6417 raft_consensus.cc:2798] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:21.442462 6417 raft_consensus.cc:491] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:21.442737 6417 raft_consensus.cc:3054] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.448997 5592 ts_tablet_manager.cc:1918] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:21.451472 4897 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 2d5c46de95bc4227bffc5e8425b8cc95 (table default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935]). Expected version 1 got 0
I20250110 20:57:21.455479 5592 log.cc:1198] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/e508061e93cc430cb88f6d10d9684f00
I20250110 20:57:21.448853 6417 raft_consensus.cc:513] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.457074 6417 leader_election.cc:290] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.457865 5592 ts_tablet_manager.cc:1939] T e508061e93cc430cb88f6d10d9684f00 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
W20250110 20:57:21.458060 4893 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a97def2f16e94061b1f938f73465fd31 consensus state has no leader
I20250110 20:57:21.458724 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 208) took 4967 ms. Trace:
I20250110 20:57:21.458985 5592 rpcz_store.cc:276] 0110 20:57:16.491352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.491478 (+ 126us) service_pool.cc:225] Handling call
0110 20:57:21.400445 (+4908967us) ts_tablet_manager.cc:1145] Deleting tablet e508061e93cc430cb88f6d10d9684f00
0110 20:57:21.400493 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.422817 (+ 22324us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.448805 (+ 25988us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.458676 (+ 9871us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4908433}
I20250110 20:57:21.461000 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f482cd146ca34f9e95b2361e9c0a985a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.461500 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 209) took 4266 ms. Trace:
I20250110 20:57:21.461606 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:21.461771 5592 rpcz_store.cc:276] 0110 20:57:17.194880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.194983 (+ 103us) service_pool.cc:225] Handling call
0110 20:57:21.460201 (+4265218us) ts_tablet_manager.cc:1145] Deleting tablet a95f05fc5afe4877a606b501dbd09551
0110 20:57:21.460275 (+ 74us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.461455 (+ 1180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7936,"tablet-delete.queue_time_us":4247602}
I20250110 20:57:21.462127 5319 raft_consensus.cc:3054] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:21.463303 5255 leader_election.cc:343] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.464725 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 219) took 4148 ms. Trace:
I20250110 20:57:21.464983 5592 rpcz_store.cc:276] 0110 20:57:17.316278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.317042 (+ 764us) service_pool.cc:225] Handling call
0110 20:57:21.464372 (+4147330us) ts_tablet_manager.cc:1145] Deleting tablet c7961f8bd3ff42ffab214a04c704fd26
0110 20:57:21.464421 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.464658 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4053713}
I20250110 20:57:21.469838 5592 tablet_replica.cc:331] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:21.470535 5592 raft_consensus.cc:2238] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:21.474390 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet c44ca47c747544b6afec83b399dcdcb5 (table default.Table_46cf92641ac847ac88303d12596df78d [id=d32dea7580494f63b0d2bab864ccc673]) successfully deleted
I20250110 20:57:21.477193 5592 raft_consensus.cc:2267] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:21.479171 5592 ts_tablet_manager.cc:1905] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:21.482887 5594 tablet_bootstrap.cc:492] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:21.483242 5594 ts_tablet_manager.cc:1397] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.202s user 0.007s sys 0.001s
I20250110 20:57:21.487504 5319 raft_consensus.cc:2463] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
W20250110 20:57:21.490054 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet a95f05fc5afe4877a606b501dbd09551 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a95f05fc5afe4877a606b501dbd09551 already in progress: creating tablet
I20250110 20:57:21.494485 5594 raft_consensus.cc:357] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.495079 5594 raft_consensus.cc:383] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.495316 5594 raft_consensus.cc:738] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.495857 5594 consensus_queue.cc:260] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.514705 5594 ts_tablet_manager.cc:1428] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.031s user 0.004s sys 0.002s
I20250110 20:57:21.514827 6104 ts_manager.cc:194] Registered new tserver with Master: 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:21.515481 5594 tablet_bootstrap.cc:492] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:21.521210 5398 leader_election.cc:304] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:21.521958 6417 raft_consensus.cc:2798] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:21.522373 6417 raft_consensus.cc:695] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:21.523047 6417 consensus_queue.cc:237] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:21.530532 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet c7961f8bd3ff42ffab214a04c704fd26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7961f8bd3ff42ffab214a04c704fd26 already in progress: creating tablet
I20250110 20:57:21.352231 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.352831 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371fb7408f0149c8b00659e09ebaf48a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:21.353423 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.368517 5773 ts_tablet_manager.cc:1905] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:21.369738 5597 ts_tablet_manager.cc:1428] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.442s user 0.003s sys 0.002s
W20250110 20:57:21.533658 5597 ts_tablet_manager.cc:1460] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0: Tablet startup took 605ms
I20250110 20:57:21.370179 6425 tablet.cc:1721] T 3d1d5fba23d8411aa7cbf4781f1c7fc1 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:21.377658 5170 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.4.171.194:57021 (request call id 135) took 1771 ms. Trace:
I20250110 20:57:21.535388 5170 rpcz_store.cc:276] 0110 20:57:19.606291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.606437 (+ 146us) service_pool.cc:225] Handling call
0110 20:57:19.727770 (+121333us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c9142 00007f8965232cd9 00007f89644c0750 00007f89645867fb 00007f8964586881 00007f896454a2a0 00007f89645357e9 00007f8964535292 00007f8962d01694
0110 20:57:19.727969 (+ 199us) raft_consensus.cc:1466] Updating replica for 1 ops
0110 20:57:19.728214 (+ 245us) raft_consensus.cc:1529] Early marking committed up to index 0
0110 20:57:19.728242 (+ 28us) raft_consensus.cc:1534] Triggering prepare for 1 ops
0110 20:57:19.728566 (+ 324us) alter_schema_op.cc:128] START. Timestamp: 1736542638624562
0110 20:57:20.787400 (+1058834us) log.cc:844] Serialized 173 byte log entry
0110 20:57:21.092092 (+304692us) raft_consensus.cc:1640] Marking committed up to 2
0110 20:57:21.365948 (+273856us) raft_consensus.cc:1692] Filling consensus response to leader.
0110 20:57:21.377326 (+ 11378us) raft_consensus.cc:1663] Waiting on the replicates to finish logging
0110 20:57:21.377371 (+ 45us) raft_consensus.cc:1680] finished
0110 20:57:21.377475 (+ 104us) raft_consensus.cc:1686] UpdateReplicas() finished
0110 20:57:21.377595 (+ 120us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:20.141498 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0110 20:57:20.142035 (+ 537us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0110 20:57:20.142065 (+ 30us) alter_schema_op.cc:66] Acquired schema lock
0110 20:57:20.142113 (+ 48us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0110 20:57:21.168449 (+1026336us) op_driver.cc:464] REPLICATION: finished
0110 20:57:21.366542 (+198093us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0110 20:57:21.535008 (+168466us) log.cc:844] Serialized 5 byte log entry
Metrics: {"spinlock_wait_cycles":276089216,"thread_start_us":200243,"threads_started":1,"wal-append.queue_time_us":187342,"child_traces":[["op",{"apply.queue_time_us":166886,"prepare.queue_time_us":412668,"prepare.run_cpu_time_us":489,"prepare.run_wall_time_us":677,"replication_time_us":1374962,"thread_start_us":1162248,"threads_started":2}]]}
I20250110 20:57:21.537843 5594 tablet_bootstrap.cc:654] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.541110 5592 ts_tablet_manager.cc:1918] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:21.541466 5592 log.cc:1198] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/01962c5b7ae64f80b998948d18250b4c
I20250110 20:57:21.542095 5592 ts_tablet_manager.cc:1939] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:21.396870 5135 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:35154 (request call id 59) took 1823 ms. Trace:
I20250110 20:57:21.401929 5136 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:35154 (request call id 58) took 1957 ms. Trace:
I20250110 20:57:21.544433 5136 rpcz_store.cc:276] 0110 20:57:19.444784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.444893 (+ 109us) service_pool.cc:225] Handling call
0110 20:57:20.845453 (+1400560us) spinlock_profiling.cc:243] Waited 711 ms on lock 0x7b5400110b54. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:21.388599 (+543146us) spinlock_profiling.cc:243] Waited 402 ms on lock 0x7b54000b40d4. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:21.401747 (+ 13148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2559477376}
I20250110 20:57:21.545785 5135 rpcz_store.cc:276] 0110 20:57:19.573553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.573661 (+ 108us) service_pool.cc:225] Handling call
0110 20:57:19.729564 (+155903us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5800001898. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f895c44a2d1 00007f895c5c9291 00007f895c5c9142 00007f8965232cd9 00007f89644c0750 00007f8964586786 00007f8964586960 00007f896453f5e2 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1
0110 20:57:20.134742 (+405178us) spinlock_profiling.cc:243] Waited 93.1 ms on lock 0x7b2000095ff8. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f896337b761 00007f896339d18c 00007f896336e732 00007f8963977482 00007f89639771ed 00007f896453f690 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02
0110 20:57:20.857907 (+723165us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b5400110b54. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:21.386363 (+528456us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b54000b40d4. stack: 00007f895c5d4853 00007f895c5d4750 00007f895c149584 00000000004fcd24 00000000004fccb9 00007f89634d46d1 00007f8963549268 00007f896453f6bd 00007f8962ddde14 00007f8962dddda2 00007f8962dddcd1 00007f8962dddc4c 00007f8962ddcf02 00007f896131cdfc 00007f896131c236 00007f896131bbdf
0110 20:57:21.396812 (+ 10449us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3136475520}
W20250110 20:57:21.534009 5597 ts_tablet_manager.cc:1462] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0: Trace:
0110 20:57:09.225970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:09.229500 (+ 3530us) service_pool.cc:225] Handling call
0110 20:57:09.230730 (+ 1230us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:09.230831 (+ 101us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:09.289978 (+ 59147us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:09.596644 (+306666us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:09.597056 (+ 412us) inbound_call.cc:173] Queueing success response
0110 20:57:19.799720 (+10202664us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0110 20:57:20.927825 (+1128105us) ts_tablet_manager.cc:1429] Starting tablet replica
0110 20:57:21.241991 (+314166us) tablet_replica.cc:260] Starting instrumentation
0110 20:57:21.242424 (+ 433us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":25997,"raft.queue_time_us":39819,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":1661,"tablet-open.queue_time_us":10202750}
I20250110 20:57:21.551012 5597 tablet_bootstrap.cc:492] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:21.430984 5110 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.195:32843, user_credentials={real_user=slave}} blocked reactor thread for 236482us
I20250110 20:57:21.443447 5222 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:21.452927 6369 raft_consensus.cc:491] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 46005a5920cc4a3697a5c61b1c271384)
I20250110 20:57:21.557169 6369 raft_consensus.cc:513] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:21.558113 5176 raft_consensus.cc:2463] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:21.558979 6369 leader_election.cc:290] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:21.557976 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 226010us
W20250110 20:57:21.561877 5399 leader_election.cc:343] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: fde470279464496aa7efdee3800754f3
I20250110 20:57:21.562242 5399 leader_election.cc:304] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:21.563624 5399 leader_election.cc:343] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:21.564327 5399 leader_election.cc:343] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.565026 5399 leader_election.cc:304] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:21.565620 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 216) took 4250 ms. Trace:
I20250110 20:57:21.565948 5592 rpcz_store.cc:276] 0110 20:57:17.315398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.390733 (+ 75335us) service_pool.cc:225] Handling call
0110 20:57:21.469701 (+4078968us) ts_tablet_manager.cc:1145] Deleting tablet 01962c5b7ae64f80b998948d18250b4c
0110 20:57:21.469746 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.496404 (+ 26658us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.540914 (+ 44510us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.565571 (+ 24657us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4036087}
I20250110 20:57:21.566365 6104 ts_manager.cc:194] Registered new tserver with Master: ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:21.568740 6415 raft_consensus.cc:2798] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:57:21.569058 6415 raft_consensus.cc:491] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 46005a5920cc4a3697a5c61b1c271384)
I20250110 20:57:21.569370 6415 raft_consensus.cc:3054] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:21.557569 5110 leader_election.cc:304] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:21.570399 6356 leader_election.cc:290] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.566649 6413 raft_consensus.cc:2743] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:21.578332 6369 raft_consensus.cc:2743] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:21.577071 6415 raft_consensus.cc:513] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
W20250110 20:57:21.582690 5398 leader_election.cc:343] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:21.584735 6415 leader_election.cc:290] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 election: Requested vote from peers 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:21.585736 5592 tablet_replica.cc:331] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:21.590339 4894 master_service.cc:946] ReplaceTablet: received request to replace tablet 51fae5f59d4642a09acee5a687b91717 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:21.589746 5594 tablet_bootstrap.cc:492] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:21.592773 5594 ts_tablet_manager.cc:1397] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.077s user 0.009s sys 0.000s
I20250110 20:57:21.594993 5594 raft_consensus.cc:357] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.595598 5594 raft_consensus.cc:383] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.595819 5594 raft_consensus.cc:738] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.602877 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc7b1acf4114bf69d801437855760df" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:21.603365 5457 raft_consensus.cc:3054] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.607117 5600 tablet_bootstrap.cc:492] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:21.607504 5600 ts_tablet_manager.cc:1397] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.224s user 0.019s sys 0.023s
I20250110 20:57:21.609529 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 01962c5b7ae64f80b998948d18250b4c (table default.Table_ff58f7ae21d3423dac4418eaa9323d72 [id=dea5264585324ac4924105c76f3a3c6f]) successfully deleted
W20250110 20:57:21.610252 5257 leader_election.cc:343] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.610630 5257 leader_election.cc:304] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.611100 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.611891 5459 raft_consensus.cc:2388] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac9003be04f340f3886b5625cc928eb0 in current term 2: Already voted for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in this term.
I20250110 20:57:21.611567 5600 raft_consensus.cc:357] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.611941 6175 raft_consensus.cc:2743] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:21.612185 5600 raft_consensus.cc:383] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.612442 5600 raft_consensus.cc:738] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.612952 5600 consensus_queue.cc:260] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.614861 5600 ts_tablet_manager.cc:1428] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20250110 20:57:21.615676 5457 raft_consensus.cc:2463] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
I20250110 20:57:21.610402 5594 consensus_queue.cc:260] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.623957 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:21.625309 5594 ts_tablet_manager.cc:1428] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250110 20:57:21.626127 5594 tablet_bootstrap.cc:492] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:21.626717 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.628930 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:21.642068 6175 raft_consensus.cc:491] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.642498 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a687769f69d844cc8564e61dc324d2b2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.642508 6175 raft_consensus.cc:513] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:21.644289 6175 leader_election.cc:290] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:21.645232 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33db8a2fdd9c4fde985345604fd04215" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.646232 5600 tablet_bootstrap.cc:492] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:21.646431 5255 leader_election.cc:343] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.664721 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f482cd146ca34f9e95b2361e9c0a985a" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.666543 5594 tablet_bootstrap.cc:654] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.675313 5600 tablet_bootstrap.cc:654] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.683466 5594 tablet_bootstrap.cc:492] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:21.683928 5594 ts_tablet_manager.cc:1397] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.000s
W20250110 20:57:21.683971 5257 leader_election.cc:343] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.684340 5257 leader_election.cc:304] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.684746 5110 leader_election.cc:304] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:21.684957 6438 raft_consensus.cc:2743] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:21.685298 5257 leader_election.cc:343] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.685721 6429 raft_consensus.cc:2798] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:21.685663 5257 leader_election.cc:304] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.686254 6438 raft_consensus.cc:2743] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:21.686560 6429 raft_consensus.cc:695] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:21.687304 6429 consensus_queue.cc:237] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.687561 5594 raft_consensus.cc:357] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.688205 5594 raft_consensus.cc:383] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.691341 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33db8a2fdd9c4fde985345604fd04215" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.576550 5597 tablet_bootstrap.cc:654] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.693449 5594 raft_consensus.cc:738] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.694113 5594 consensus_queue.cc:260] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.695873 5594 ts_tablet_manager.cc:1428] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20250110 20:57:21.696666 5594 tablet_bootstrap.cc:492] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:21.698546 6425 tablet.cc:1721] T 2d5c46de95bc4227bffc5e8425b8cc95 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:21.706991 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6176c29f01b4ca5888c844c131270f2" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:21.707880 5600 tablet_bootstrap.cc:492] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:21.708259 5600 ts_tablet_manager.cc:1397] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.062s user 0.006s sys 0.002s
I20250110 20:57:21.708299 5176 raft_consensus.cc:3054] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:21.710307 5600 raft_consensus.cc:357] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.710949 5600 raft_consensus.cc:383] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.711215 5600 raft_consensus.cc:738] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.711833 5600 consensus_queue.cc:260] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.713899 5600 ts_tablet_manager.cc:1428] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250110 20:57:21.717586 5594 tablet_bootstrap.cc:654] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:21.718209 5399 leader_election.cc:343] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Not found: Tablet not found: 01962c5b7ae64f80b998948d18250b4c
I20250110 20:57:21.718572 5399 leader_election.cc:304] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:21.719059 5399 raft_consensus.cc:2681] T 01962c5b7ae64f80b998948d18250b4c P 3c5bff1e70804ab4a1bb63bdb9345ce6: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250110 20:57:21.720834 5399 leader_election.cc:343] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:21.723743 5257 leader_election.cc:343] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.724078 5257 leader_election.cc:304] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.724483 5600 tablet_bootstrap.cc:492] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:21.724489 5399 leader_election.cc:343] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.725198 6438 raft_consensus.cc:2743] T 33db8a2fdd9c4fde985345604fd04215 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:21.728128 5399 leader_election.cc:343] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.740247 5600 tablet_bootstrap.cc:654] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.743481 5594 tablet_bootstrap.cc:492] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:21.743851 5594 ts_tablet_manager.cc:1397] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.001s
I20250110 20:57:21.745998 5594 raft_consensus.cc:357] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.746609 5594 raft_consensus.cc:383] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.746850 5594 raft_consensus.cc:738] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.747355 5594 consensus_queue.cc:260] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.748828 5594 ts_tablet_manager.cc:1428] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250110 20:57:21.753392 5594 tablet_bootstrap.cc:492] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:21.758271 5594 tablet_bootstrap.cc:654] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.760138 5773 ts_tablet_manager.cc:1918] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:21.769079 5592 raft_consensus.cc:2238] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:21.769584 5592 raft_consensus.cc:2267] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:21.771250 5592 ts_tablet_manager.cc:1905] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:21.772586 5600 tablet_bootstrap.cc:492] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:21.772938 5600 ts_tablet_manager.cc:1397] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.049s user 0.010s sys 0.002s
I20250110 20:57:21.793808 5600 raft_consensus.cc:357] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.794457 5600 raft_consensus.cc:383] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.794693 5600 raft_consensus.cc:738] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.795233 5600 consensus_queue.cc:260] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.814349 4892 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36568 (request call id 318) took 1276 ms. Trace:
I20250110 20:57:21.816159 5600 ts_tablet_manager.cc:1428] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.043s user 0.003s sys 0.003s
I20250110 20:57:21.816932 5600 tablet_bootstrap.cc:492] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:21.814677 4892 rpcz_store.cc:276] 0110 20:57:20.537502 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.537625 (+ 123us) service_pool.cc:225] Handling call
0110 20:57:20.669276 (+131651us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b306 00007f3bfcfff044 00007f3bfcffefd2 00007f3bfcffef01 00007f3bfcffee7c 00007f3bfcffe132 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:20.671211 (+ 1935us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:20.672560 (+ 1349us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:21.813496 (+1140936us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:21.813860 (+ 364us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:21.814180 (+ 320us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:21.203557 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:21.203906 (+ 349us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:21.203929 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:57:21.203943 (+ 14us) tablet.cc:659] Decoding operations
0110 20:57:21.204216 (+ 273us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:21.204252 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:21.204275 (+ 23us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:21.204468 (+ 193us) tablet.cc:698] Row locks acquired
0110 20:57:21.204482 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:21.204595 (+ 113us) write_op.cc:270] Start()
0110 20:57:21.204680 (+ 85us) write_op.cc:276] Timestamp: P: 1736542641204503 usec, L: 0
0110 20:57:21.204695 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:21.205224 (+ 529us) log.cc:844] Serialized 433 byte log entry
0110 20:57:21.807368 (+602144us) op_driver.cc:464] REPLICATION: finished
0110 20:57:21.808238 (+ 870us) write_op.cc:301] APPLY: starting
0110 20:57:21.808300 (+ 62us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:21.808399 (+ 99us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:21.808493 (+ 94us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:21.808869 (+ 376us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:21.808994 (+ 125us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:21.809033 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:57:21.809197 (+ 164us) log.cc:844] Serialized 29 byte log entry
0110 20:57:21.809329 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.809428 (+ 99us) write_op.cc:454] Released schema lock
0110 20:57:21.809458 (+ 30us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:21.809685 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.809709 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":296491776,"child_traces":[["op",{"apply.queue_time_us":709,"apply.run_cpu_time_us":1428,"apply.run_wall_time_us":1428,"num_ops":2,"prepare.queue_time_us":524641,"prepare.run_cpu_time_us":1885,"prepare.run_wall_time_us":21087,"replication_time_us":602576}]]}
I20250110 20:57:21.818113 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet adfed905cadb4e82acd3ff066c8c4cba
I20250110 20:57:21.819427 4900 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 319) took 1247 ms. Trace:
I20250110 20:57:21.821379 5600 tablet_bootstrap.cc:654] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:21.821700 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet adfed905cadb4e82acd3ff066c8c4cba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:21.819679 4900 rpcz_store.cc:276] 0110 20:57:20.571606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.571733 (+ 127us) service_pool.cc:225] Handling call
0110 20:57:20.665203 (+ 93470us) spinlock_profiling.cc:243] Waited 92.8 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b616 00007f3bfd003b44 00007f3bfd003ad2 00007f3bfd003a01 00007f3bfd00397c 00007f3bfd002c32 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:20.666109 (+ 906us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:20.666615 (+ 506us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:20.666714 (+ 99us) catalog_manager.cc:2762] Locking tablets
0110 20:57:20.958166 (+291452us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:21.817575 (+859409us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:21.817846 (+ 271us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:21.817954 (+ 108us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:21.819358 (+ 1404us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:21.224771 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:21.225138 (+ 367us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:21.225160 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:21.225171 (+ 11us) tablet.cc:659] Decoding operations
0110 20:57:21.225489 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:21.225522 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:21.252420 (+ 26898us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:21.252680 (+ 260us) tablet.cc:698] Row locks acquired
0110 20:57:21.252694 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:21.252804 (+ 110us) write_op.cc:270] Start()
0110 20:57:21.252886 (+ 82us) write_op.cc:276] Timestamp: P: 1736542641252708 usec, L: 0
0110 20:57:21.252901 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:21.253351 (+ 450us) log.cc:844] Serialized 731 byte log entry
0110 20:57:21.807581 (+554230us) op_driver.cc:464] REPLICATION: finished
0110 20:57:21.809915 (+ 2334us) write_op.cc:301] APPLY: starting
0110 20:57:21.809979 (+ 64us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:21.810083 (+ 104us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:21.810177 (+ 94us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:21.810420 (+ 243us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:21.810543 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:21.810582 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:57:21.810740 (+ 158us) log.cc:844] Serialized 29 byte log entry
0110 20:57:21.810853 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.810916 (+ 63us) write_op.cc:454] Released schema lock
0110 20:57:21.810942 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:21.811136 (+ 194us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.811177 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":213402368,"child_traces":[["op",{"apply.queue_time_us":2221,"apply.run_cpu_time_us":1175,"apply.run_wall_time_us":1175,"num_ops":2,"prepare.queue_time_us":265086,"prepare.run_cpu_time_us":1965,"prepare.run_wall_time_us":31374,"replication_time_us":554601}]]}
I20250110 20:57:21.839437 4900 master_service.cc:946] ReplaceTablet: received request to replace tablet 637a39e4264b40dd9480dc13e37359d9 from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:21.844938 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet adfed905cadb4e82acd3ff066c8c4cba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:21.873530 6438 raft_consensus.cc:491] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.874042 6438 raft_consensus.cc:513] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.875787 6438 leader_election.cc:290] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:21.877804 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdb2ee9419244a9ab939be56b2d7c72" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.878947 6451 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b84772eba3f0460faf8523f76eb57e53)
W20250110 20:57:21.879585 5255 leader_election.cc:343] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.879359 6451 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:21.882380 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:57:21.888408 5592 ts_tablet_manager.cc:1918] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:21.889536 6419 raft_consensus.cc:491] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.889915 6419 raft_consensus.cc:513] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.891749 6419 leader_election.cc:290] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.893260 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb75e1886e74a719dc09ccad8e1e284" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:21.893869 5318 raft_consensus.cc:2463] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:21.898016 5398 leader_election.cc:304] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
W20250110 20:57:21.766224 5111 leader_election.cc:343] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:21.901902 5111 leader_election.cc:304] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:21.902745 5773 log.cc:1198] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/527bbddda8184a6eb67c2411ae50db63
I20250110 20:57:21.903482 5773 ts_tablet_manager.cc:1939] T 527bbddda8184a6eb67c2411ae50db63 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:21.905653 6419 raft_consensus.cc:2798] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:21.905956 6419 raft_consensus.cc:491] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:21.906232 6419 raft_consensus.cc:3054] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:21.908310 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet adfed905cadb4e82acd3ff066c8c4cba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adfed905cadb4e82acd3ff066c8c4cba already in progress: creating tablet
I20250110 20:57:21.885006 5597 tablet_bootstrap.cc:492] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:21.884294 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet adfed905cadb4e82acd3ff066c8c4cba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:21.884505 6369 raft_consensus.cc:491] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.909643 6369 raft_consensus.cc:513] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.912237 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:21.884626 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fdb2ee9419244a9ab939be56b2d7c72" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.912979 4891 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:36568 (request call id 322) took 1048 ms. Trace:
I20250110 20:57:21.914057 5597 ts_tablet_manager.cc:1397] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.363s user 0.009s sys 0.000s
I20250110 20:57:21.913250 4891 rpcz_store.cc:276] 0110 20:57:20.864054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.874040 (+ 9986us) service_pool.cc:225] Handling call
0110 20:57:21.032096 (+158056us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b306 00007f3bfcfff044 00007f3bfcffefd2 00007f3bfcffef01 00007f3bfcffee7c 00007f3bfcffe132 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:21.061633 (+ 29537us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:21.062977 (+ 1344us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:21.912060 (+849083us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:21.912484 (+ 424us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:21.912826 (+ 342us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:21.261926 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:21.262243 (+ 317us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:21.262264 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:57:21.262276 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:21.263748 (+ 1472us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:21.263780 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:21.263801 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:21.290179 (+ 26378us) tablet.cc:698] Row locks acquired
0110 20:57:21.290195 (+ 16us) write_op.cc:260] PREPARE: finished
0110 20:57:21.290313 (+ 118us) write_op.cc:270] Start()
0110 20:57:21.290395 (+ 82us) write_op.cc:276] Timestamp: P: 1736542641290208 usec, L: 0
0110 20:57:21.290409 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:21.292584 (+ 2175us) log.cc:844] Serialized 433 byte log entry
0110 20:57:21.909353 (+616769us) op_driver.cc:464] REPLICATION: finished
0110 20:57:21.910061 (+ 708us) write_op.cc:301] APPLY: starting
0110 20:57:21.910132 (+ 71us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:21.910253 (+ 121us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:21.910342 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:21.910730 (+ 388us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:21.910850 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:21.910887 (+ 37us) write_op.cc:312] APPLY: finished
0110 20:57:21.911142 (+ 255us) log.cc:844] Serialized 29 byte log entry
0110 20:57:21.911688 (+ 546us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.911756 (+ 68us) write_op.cc:454] Released schema lock
0110 20:57:21.911784 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:21.912005 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.912030 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":354342784,"child_traces":[["op",{"apply.queue_time_us":304,"apply.run_cpu_time_us":1441,"apply.run_wall_time_us":1895,"num_ops":2,"prepare.queue_time_us":197870,"prepare.run_cpu_time_us":1851,"prepare.run_wall_time_us":32856,"replication_time_us":618847}]]}
W20250110 20:57:21.917095 5257 leader_election.cc:343] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.917461 5257 leader_election.cc:304] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:21.918038 6438 raft_consensus.cc:2743] T 3fdb2ee9419244a9ab939be56b2d7c72 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:21.916116 5597 raft_consensus.cc:357] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.922341 6369 leader_election.cc:290] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:21.922669 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 156463us
I20250110 20:57:21.931434 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:57:21.933059 6034 leader_election.cc:304] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14d1e2d5c72f498f915c367e8750361f; no voters: 63aa77ee6ec84a6e82132b2de9e77486, b84772eba3f0460faf8523f76eb57e53
I20250110 20:57:21.934309 6413 raft_consensus.cc:491] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:21.934720 6413 raft_consensus.cc:513] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.937091 6413 leader_election.cc:290] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.938076 6459 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:57:21.939342 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c87076a8a4b1e86f7e4844e17b1f3" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:21.940006 5318 raft_consensus.cc:2463] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:21.941213 5398 leader_election.cc:304] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:21.941987 6413 raft_consensus.cc:2798] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:21.942350 6413 raft_consensus.cc:491] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:21.942665 6413 raft_consensus.cc:3054] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.944111 4892 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_094c089272f6475a989a44b5e1843fd5" } new_table_name: "default.Table_01a45b23c3694bd991e5a47e493e3b4d" modify_external_catalogs: true
I20250110 20:57:21.941272 4898 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_7b913fb45b9d478ba496dbf9d6703f30" } new_table_name: "default.Table_54bc21f0baba4cc4b8db0325939c0427" modify_external_catalogs: true
I20250110 20:57:21.947988 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee23ab7ecce465ca1153d22c40bf935" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:21.951165 5170 raft_consensus.cc:3054] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.960774 6419 raft_consensus.cc:513] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.962046 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb75e1886e74a719dc09ccad8e1e284" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.963719 6419 leader_election.cc:290] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:21.963927 5320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb75e1886e74a719dc09ccad8e1e284" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:21.964450 5320 raft_consensus.cc:3054] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:21.968690 6458 tablet.cc:1721] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:21.968950 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c87076a8a4b1e86f7e4844e17b1f3" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:21.969681 6451 leader_election.cc:290] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381)
I20250110 20:57:21.970672 5597 raft_consensus.cc:383] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.970943 5597 raft_consensus.cc:738] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.971189 5320 raft_consensus.cc:2463] T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:21.971927 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5eb75e1886e74a719dc09ccad8e1e284" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:21.972195 5398 leader_election.cc:304] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:21.973172 6417 raft_consensus.cc:2798] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:21.973627 6417 raft_consensus.cc:695] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:21.974437 6417 consensus_queue.cc:237] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.974608 6457 tablet.cc:1721] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:21.888743 5592 log.cc:1198] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/fde470279464496aa7efdee3800754f3
I20250110 20:57:21.977936 5592 ts_tablet_manager.cc:1939] T fde470279464496aa7efdee3800754f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:21.988397 5594 tablet_bootstrap.cc:492] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:21.988777 5594 ts_tablet_manager.cc:1397] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.236s user 0.031s sys 0.043s
I20250110 20:57:21.990923 5594 raft_consensus.cc:357] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.991547 5594 raft_consensus.cc:383] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:21.991783 5594 raft_consensus.cc:738] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:21.991847 4894 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 51fae5f59d4642a09acee5a687b91717
W20250110 20:57:21.991940 5399 leader_election.cc:343] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.992269 5594 consensus_queue.cc:260] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:21.993060 4894 catalog_manager.cc:6596] ReplaceTablet: tablet 51fae5f59d4642a09acee5a687b91717 deleted and replaced by tablet 30bff8d59a484f878bb61937e1e5f3f8
W20250110 20:57:21.993142 5399 leader_election.cc:343] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.994196 5594 ts_tablet_manager.cc:1428] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20250110 20:57:21.994984 5594 tablet_bootstrap.cc:492] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
W20250110 20:57:21.994931 5399 leader_election.cc:343] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:21.995081 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fae5f59d4642a09acee5a687b91717 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bff8d59a484f878bb61937e1e5f3f8) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:21.996625 5443 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fae5f59d4642a09acee5a687b91717 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bff8d59a484f878bb61937e1e5f3f8) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:21.998817 4892 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33db8a2fdd9c4fde985345604fd04215 consensus state has no leader
I20250110 20:57:22.000458 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet 51fae5f59d4642a09acee5a687b91717 with delete_type TABLET_DATA_DELETED (replaced by tablet 30bff8d59a484f878bb61937e1e5f3f8) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:22.001895 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 51fae5f59d4642a09acee5a687b91717 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51fae5f59d4642a09acee5a687b91717 already in progress: creating tablet
I20250110 20:57:21.924353 5176 raft_consensus.cc:2463] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 2.
I20250110 20:57:22.005782 6429 raft_consensus.cc:2743] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:22.024134 6388 consensus_queue.cc:1035] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:22.033088 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 224) took 4495 ms. Trace:
I20250110 20:57:22.035069 5597 consensus_queue.cc:260] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.038455 6413 raft_consensus.cc:513] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.047607 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_3d09af5dec5b4fcebdd16b2dcdd03e6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.048988 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d09af5dec5b4fcebdd16b2dcdd03e6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.050664 6369 raft_consensus.cc:491] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.051059 6369 raft_consensus.cc:513] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.054226 6369 leader_election.cc:290] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:22.058477 5597 ts_tablet_manager.cc:1428] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.144s user 0.004s sys 0.000s
I20250110 20:57:22.060271 4892 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2d69d3afb9ca476cb5b329145ef8f74c" } modify_external_catalogs: true
I20250110 20:57:22.060824 4892 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_2d69d3afb9ca476cb5b329145ef8f74c" } modify_external_catalogs: true
I20250110 20:57:22.062737 5597 tablet_bootstrap.cc:492] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:22.042369 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 236) took 4022 ms. Trace:
I20250110 20:57:22.065538 5773 rpcz_store.cc:276] 0110 20:57:18.019629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.019736 (+ 107us) service_pool.cc:225] Handling call
0110 20:57:21.116493 (+3096757us) ts_tablet_manager.cc:1145] Deleting tablet 527bbddda8184a6eb67c2411ae50db63
0110 20:57:21.116530 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.634923 (+518393us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.759938 (+125015us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.042309 (+282371us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":247842,"tablet-delete.queue_time_us":3078874}
I20250110 20:57:22.067129 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 527bbddda8184a6eb67c2411ae50db63 (table default.Table_a3a9cc7de02e4fba887c5bc79ffd6713 [id=8ae7ea615d674c8faece968ce0c4696a]) successfully deleted
W20250110 20:57:22.040115 5111 consensus_peers.cc:487] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 637a39e4264b40dd9480dc13e37359d9. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:22.059660 5110 leader_election.cc:343] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Not found: Tablet not found: e508061e93cc430cb88f6d10d9684f00
I20250110 20:57:22.079752 5111 leader_election.cc:304] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:22.084525 6461 tablet.cc:1721] T f2cbb46ea11c4e7d8bc5993ccdc52247 P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:22.093529 6369 raft_consensus.cc:2743] T eee23ab7ecce465ca1153d22c40bf935 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.097340 5600 tablet_bootstrap.cc:492] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:22.097765 5600 ts_tablet_manager.cc:1397] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.281s user 0.027s sys 0.013s
I20250110 20:57:22.100100 5600 raft_consensus.cc:357] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.100705 5600 raft_consensus.cc:383] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:22.100930 5600 raft_consensus.cc:738] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:22.101548 5600 consensus_queue.cc:260] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.103528 5773 tablet_replica.cc:331] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:22.104208 5773 raft_consensus.cc:2238] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:22.104691 5773 raft_consensus.cc:2267] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:22.108954 5600 ts_tablet_manager.cc:1428] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250110 20:57:22.109748 5600 tablet_bootstrap.cc:492] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:22.040886 5592 rpcz_store.cc:276] 0110 20:57:17.537762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.538423 (+ 661us) service_pool.cc:225] Handling call
0110 20:57:21.567959 (+4029536us) ts_tablet_manager.cc:1145] Deleting tablet fde470279464496aa7efdee3800754f3
0110 20:57:21.585629 (+ 17670us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:21.777953 (+192324us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:21.888091 (+110138us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.033032 (+144941us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":179570,"tablet-delete.queue_time_us":4029042}
I20250110 20:57:22.081359 5399 leader_election.cc:304] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:22.112847 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet fde470279464496aa7efdee3800754f3 (table default.Table_5d22ef7f33a3452db4b509894cf54587 [id=4ec4b51d6cbb45929fbce18b91226b1a]) successfully deleted
I20250110 20:57:22.113976 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 229) took 4193 ms. Trace:
I20250110 20:57:22.114326 5592 rpcz_store.cc:276] 0110 20:57:17.920635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:17.974072 (+ 53437us) service_pool.cc:225] Handling call
0110 20:57:22.113582 (+4139510us) ts_tablet_manager.cc:1145] Deleting tablet 32cf221df30e4e78bd2a8078c26d29ba
0110 20:57:22.113646 (+ 64us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.113928 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4138843}
W20250110 20:57:22.115569 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 32cf221df30e4e78bd2a8078c26d29ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32cf221df30e4e78bd2a8078c26d29ba already in progress: creating tablet
I20250110 20:57:22.120337 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_42c9aae4bec849858299f5a64a730abb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.121747 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42c9aae4bec849858299f5a64a730abb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.130877 4891 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a814f17e793b48a3b619651e11d38708"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.132287 4891 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a814f17e793b48a3b619651e11d38708 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.133970 6205 raft_consensus.cc:2798] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:57:22.134590 6356 raft_consensus.cc:491] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.134958 6356 raft_consensus.cc:513] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.136624 6356 leader_election.cc:290] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:22.134419 6205 raft_consensus.cc:695] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 2 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:22.139154 6205 consensus_queue.cc:237] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } }
I20250110 20:57:22.144325 4894 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_8f7a6268867c4d31b49ef33283cf5c6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.149648 4894 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f7a6268867c4d31b49ef33283cf5c6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.116703 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 230) took 3550 ms. Trace:
I20250110 20:57:22.160223 5592 rpcz_store.cc:276] 0110 20:57:18.566153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.566253 (+ 100us) service_pool.cc:225] Handling call
0110 20:57:22.116310 (+3550057us) ts_tablet_manager.cc:1145] Deleting tablet 4b28befa048244309a06f69b8a3b4b8f
0110 20:57:22.116367 (+ 57us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.116647 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3549600}
W20250110 20:57:22.164245 4885 catalog_manager.cc:4852] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 4b28befa048244309a06f69b8a3b4b8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b28befa048244309a06f69b8a3b4b8f already in progress: creating tablet
I20250110 20:57:22.179008 6356 raft_consensus.cc:491] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 20:57:22.179154 5398 leader_election.cc:343] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: bae52ea080f54d228b6b76922248ed5a
I20250110 20:57:22.179450 6356 raft_consensus.cc:513] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.181172 6356 leader_election.cc:290] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:22.188354 5398 leader_election.cc:343] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: f3104928188c489d93bfd499d148fa5d
I20250110 20:57:22.195137 5592 tablet_replica.cc:331] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:22.195626 6413 leader_election.cc:290] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:22.195814 5592 raft_consensus.cc:2238] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 2 LEADER]: Raft consensus shutting down.
I20250110 20:57:22.196427 5592 pending_rounds.cc:62] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:22.196626 5592 pending_rounds.cc:69] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7112878662209638400 op_type: NO_OP noop_request { }
I20250110 20:57:22.196930 5592 raft_consensus.cc:2883] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:22.197193 5592 raft_consensus.cc:2267] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:22.202968 5773 ts_tablet_manager.cc:1905] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:22.208037 5175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c87076a8a4b1e86f7e4844e17b1f3" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
I20250110 20:57:22.153276 5600 tablet_bootstrap.cc:654] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:22.209215 5171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3104928188c489d93bfd499d148fa5d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:22.213819 4891 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_f88b13f73f784b8890702e20bc102be2" } new_table_name: "default.Table_290967aa2a704b0ca28eb1ffb2870832" modify_external_catalogs: true
W20250110 20:57:22.216784 5399 consensus_peers.cc:487] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:22.217345 5399 consensus_peers.cc:487] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:22.218984 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8c87076a8a4b1e86f7e4844e17b1f3" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:22.219548 5319 raft_consensus.cc:3054] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:22.224263 6438 raft_consensus.cc:491] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.224653 6438 raft_consensus.cc:513] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.226573 6438 leader_election.cc:290] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:22.230226 5319 raft_consensus.cc:2463] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:22.231349 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bae52ea080f54d228b6b76922248ed5a" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:22.235006 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet f80685cb3c494283bea421195bf7751f (table default.Table_3f641bf1cdfd46fda1bd487fa4f01f1e [id=75dcee4b73484274bac56a2ab1a827d7]). Expected version 1 got 0
I20250110 20:57:22.236508 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet a68d4b44975b436fb1b4a9ee6ff290d2 (table default.Table_fd872d12446c4621a0aaad2289d63e1d [id=9a3db504db8449efb89996c55376ab19]). Expected version 2 got 0
I20250110 20:57:22.237391 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 3fdb2ee9419244a9ab939be56b2d7c72 (table default.Table_fd1dd024f4824116bfc0ceb0584e3f8a [id=af1b518112b74eb2a7568ddb2e8a1935]). Expected version 3 got 0
I20250110 20:57:22.237957 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 33db8a2fdd9c4fde985345604fd04215 (table default.Table_01a45b23c3694bd991e5a47e493e3b4d [id=505b9ca1a68e401aa6f6f28a41f09181]). Expected version 2 got 0
I20250110 20:57:22.238580 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 76bedccd828042948a702fe840788765 (table default.Table_b2bea6d7461644019524e9f4455cde2e [id=bef02f1acb0e4268882c92bc6c3618b9]). Expected version 1 got 0
I20250110 20:57:22.227897 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:22.242337 5255 leader_election.cc:343] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.244997 5592 ts_tablet_manager.cc:1905] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:22.244050 4893 catalog_manager.cc:5526] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:22.245761 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:22.246454 4893 catalog_manager.cc:5526] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:22.247613 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_7b913fb45b9d478ba496dbf9d6703f30 [id=8fab9a2044704cacb68b9c16330ff3ff]). Expected version 1 got 0
I20250110 20:57:22.248384 4895 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_85d9be9fb9414103a1623ee3dbe714ae [id=79b8eb01b2344a4897e1bc773e7d7935] alter complete (version 1)
I20250110 20:57:22.251103 5398 leader_election.cc:304] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
W20250110 20:57:22.251497 4895 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:58569 (request call id 70) took 1276 ms (client timeout 1000 ms). Trace:
I20250110 20:57:22.252005 6464 raft_consensus.cc:2798] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:22.252431 6464 raft_consensus.cc:695] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:22.253238 6464 consensus_queue.cc:237] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:22.256537 4897 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:46155 (request call id 48) took 850 ms (client timeout 1000 ms). Trace:
W20250110 20:57:22.256809 4897 rpcz_store.cc:269] 0110 20:57:21.405951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:21.406069 (+ 118us) service_pool.cc:225] Handling call
0110 20:57:22.256479 (+850410us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
W20250110 20:57:22.251786 4895 rpcz_store.cc:269] 0110 20:57:20.975503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.975623 (+ 120us) service_pool.cc:225] Handling call
0110 20:57:21.040840 (+ 65217us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff299dee 00007f3bfcff7fc4 00007f3bfcff7f52 00007f3bfcff7e81 00007f3bfcff7dfc 00007f3bfcff70b2 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:22.251418 (+1210578us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:21.393151 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:21.393563 (+ 412us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:21.393596 (+ 33us) write_op.cc:435] Acquired schema lock
0110 20:57:21.393615 (+ 19us) tablet.cc:659] Decoding operations
0110 20:57:21.393883 (+ 268us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:21.393930 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:21.393950 (+ 20us) tablet.cc:682] Acquiring locks for 1 operations
0110 20:57:21.394117 (+ 167us) tablet.cc:698] Row locks acquired
0110 20:57:21.394137 (+ 20us) write_op.cc:260] PREPARE: finished
0110 20:57:21.394249 (+ 112us) write_op.cc:270] Start()
0110 20:57:21.394344 (+ 95us) write_op.cc:276] Timestamp: P: 1736542641394132 usec, L: 0
0110 20:57:21.394376 (+ 32us) op_driver.cc:348] REPLICATION: starting
0110 20:57:21.395042 (+ 666us) log.cc:844] Serialized 356 byte log entry
0110 20:57:21.918389 (+523347us) op_driver.cc:464] REPLICATION: finished
0110 20:57:21.918758 (+ 369us) write_op.cc:301] APPLY: starting
0110 20:57:21.918807 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:21.918887 (+ 80us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:21.918979 (+ 92us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:21.919142 (+ 163us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:21.919272 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0110 20:57:21.919312 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:21.919450 (+ 138us) log.cc:844] Serialized 23 byte log entry
0110 20:57:21.920043 (+ 593us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.920099 (+ 56us) write_op.cc:454] Released schema lock
0110 20:57:21.920128 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:21.920442 (+ 314us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.920469 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":145060096,"child_traces":[["op",{"apply.queue_time_us":232,"apply.run_cpu_time_us":1110,"apply.run_wall_time_us":1651,"num_ops":1,"prepare.queue_time_us":44923,"prepare.run_cpu_time_us":2157,"prepare.run_wall_time_us":2158,"replication_time_us":523934}]]}
I20250110 20:57:22.261642 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet a687769f69d844cc8564e61dc324d2b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:12 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:22.263152 4898 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2644e9180a9548c08cf9e7c4bebb8189 consensus state has no leader
I20250110 20:57:22.263769 5149 tablet_service.cc:1467] Processing CreateTablet for tablet c5363ad585b549bf8def9f28a76e526e (DEFAULT_TABLE table=default.Table_06885b5a5dd84257ac0f7be72678888f [id=c37b93e32ad94b4385deab24175f4612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.263995 4899 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 432078ac2fe347e1a55834ca0f77531a
I20250110 20:57:22.264765 5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5363ad585b549bf8def9f28a76e526e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.265241 4899 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:36568 (request call id 321) took 1685 ms. Trace:
I20250110 20:57:22.265679 5293 tablet_service.cc:1467] Processing CreateTablet for tablet c5363ad585b549bf8def9f28a76e526e (DEFAULT_TABLE table=default.Table_06885b5a5dd84257ac0f7be72678888f [id=c37b93e32ad94b4385deab24175f4612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.266556 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5363ad585b549bf8def9f28a76e526e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.267870 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet 527bbddda8184a6eb67c2411ae50db63 with delete_type TABLET_DATA_DELETED (replaced by tablet 9633eb4580c04f11a48ff5055e257c50) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:22.268541 6454 tablet_replica.cc:331] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:22.269272 6454 raft_consensus.cc:2238] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:22.269695 6454 raft_consensus.cc:2267] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:22.270376 6401 catalog_manager.cc:1260] Loaded cluster ID: 9189b87cd4594c99bd9b29530bf32da1
I20250110 20:57:22.270612 6401 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: loading cluster ID for follower catalog manager: success
I20250110 20:57:22.271333 6454 ts_tablet_manager.cc:1905] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:22.275969 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet 432078ac2fe347e1a55834ca0f77531a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:22.276875 5443 tablet_service.cc:1467] Processing CreateTablet for tablet c5363ad585b549bf8def9f28a76e526e (DEFAULT_TABLE table=default.Table_06885b5a5dd84257ac0f7be72678888f [id=c37b93e32ad94b4385deab24175f4612]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.277766 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5363ad585b549bf8def9f28a76e526e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.283943 5290 tablet_service.cc:1514] Processing DeleteTablet for tablet 432078ac2fe347e1a55834ca0f77531a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:22.285322 6419 consensus_queue.cc:1035] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:22.285151 5152 tablet_service.cc:1514] Processing DeleteTablet for tablet 432078ac2fe347e1a55834ca0f77531a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:20 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:22.292975 5152 tablet_service.cc:1467] Processing CreateTablet for tablet eaa58168a2bf4b1a886b12ac98314bbc (DEFAULT_TABLE table=default.Table_50279f2aee724e4792bbc5ccba5d0560 [id=e531b085c893487ead13efcf1f9cfa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.293983 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa58168a2bf4b1a886b12ac98314bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.297240 5290 tablet_service.cc:1467] Processing CreateTablet for tablet eaa58168a2bf4b1a886b12ac98314bbc (DEFAULT_TABLE table=default.Table_50279f2aee724e4792bbc5ccba5d0560 [id=e531b085c893487ead13efcf1f9cfa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.298738 5290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa58168a2bf4b1a886b12ac98314bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.299353 5444 tablet_service.cc:1467] Processing CreateTablet for tablet eaa58168a2bf4b1a886b12ac98314bbc (DEFAULT_TABLE table=default.Table_50279f2aee724e4792bbc5ccba5d0560 [id=e531b085c893487ead13efcf1f9cfa0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.300045 6417 consensus_queue.cc:1035] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:22.301568 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef432f354774806adf44ed76195856f (DEFAULT_TABLE table=default.Table_2a752f83903946dea40ca65dd07107ce [id=156e2135b3ff4411bf2ebf47e6233363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.302508 5138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef432f354774806adf44ed76195856f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.303308 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaa58168a2bf4b1a886b12ac98314bbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.265519 4899 rpcz_store.cc:276] 0110 20:57:20.580104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.580588 (+ 484us) service_pool.cc:225] Handling call
0110 20:57:20.661336 (+ 80748us) spinlock_profiling.cc:243] Waited 80.1 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd479d50 00007f3bfd4fcf2a 00007f3bff1a1dec 00007f3bff29b616 00007f3bfd003b44 00007f3bfd003ad2 00007f3bfd003a01 00007f3bfd00397c 00007f3bfd002c32 00007f3bfb2c1dfc 00007f3bfb2c1236
0110 20:57:20.667552 (+ 6216us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:20.668042 (+ 490us) catalog_manager.cc:2753] Modifying in-memory table state
0110 20:57:20.668148 (+ 106us) catalog_manager.cc:2762] Locking tablets
0110 20:57:20.956367 (+288219us) catalog_manager.cc:2776] Removing table and tablets from system table
0110 20:57:21.842664 (+886297us) catalog_manager.cc:2794] Removing table from by-name map
0110 20:57:21.842910 (+ 246us) catalog_manager.cc:2810] Committing in-memory state
0110 20:57:22.263832 (+420922us) catalog_manager.cc:2814] Aborting table tasks
0110 20:57:22.265189 (+ 1357us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:21.256273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:21.256603 (+ 330us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:21.256623 (+ 20us) write_op.cc:435] Acquired schema lock
0110 20:57:21.256636 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:21.258557 (+ 1921us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:21.258589 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:21.258620 (+ 31us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:21.258846 (+ 226us) tablet.cc:698] Row locks acquired
0110 20:57:21.258859 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:21.258957 (+ 98us) write_op.cc:270] Start()
0110 20:57:21.259029 (+ 72us) write_op.cc:276] Timestamp: P: 1736542641258862 usec, L: 0
0110 20:57:21.259044 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:21.259486 (+ 442us) log.cc:844] Serialized 731 byte log entry
0110 20:57:21.807774 (+548288us) op_driver.cc:464] REPLICATION: finished
0110 20:57:21.811404 (+ 3630us) write_op.cc:301] APPLY: starting
0110 20:57:21.811456 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:21.811558 (+ 102us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:21.811633 (+ 75us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:21.811847 (+ 214us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:21.811963 (+ 116us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:21.812008 (+ 45us) write_op.cc:312] APPLY: finished
0110 20:57:21.812162 (+ 154us) log.cc:844] Serialized 29 byte log entry
0110 20:57:21.812270 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:21.812334 (+ 64us) write_op.cc:454] Released schema lock
0110 20:57:21.812360 (+ 26us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":184149888,"child_traces":[["op",{"apply.queue_time_us":3527,"apply.run_cpu_time_us":1133,"apply.run_wall_time_us":31634,"num_ops":2,"prepare.queue_time_us":292010,"prepare.run_cpu_time_us":1772,"prepare.run_wall_time_us":5535,"replication_time_us":548639,"thread_start_us":101102,"threads_started":1}]]}
I20250110 20:57:22.313694 6438 raft_consensus.cc:491] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.314119 6438 raft_consensus.cc:513] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.315838 6438 leader_election.cc:290] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:22.320819 6461 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.235s user 0.001s sys 0.000s
I20250110 20:57:22.321286 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72bb739d12244eca51224bc27f0ed1c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:22.322650 5255 leader_election.cc:343] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.327833 5141 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6b42a01e4944dfa0f2691d4bb86446 (DEFAULT_TABLE table=default.Table_91296c9c7dfb47dab70fcfd591e7f6b4 [id=bd31ccb0c83f40ed80252496b582e79e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.329849 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 27ee4c158df747bab2c1e97a12d86f14 (DEFAULT_TABLE table=default.Table_fce3f8bf894845e287acf83b2e2fe8c2 [id=d6e67381be374d248c8bdd8723b7a83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.329891 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6b42a01e4944dfa0f2691d4bb86446 (DEFAULT_TABLE table=default.Table_91296c9c7dfb47dab70fcfd591e7f6b4 [id=bd31ccb0c83f40ed80252496b582e79e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.330754 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b6b42a01e4944dfa0f2691d4bb86446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.330957 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6b42a01e4944dfa0f2691d4bb86446 (DEFAULT_TABLE table=default.Table_91296c9c7dfb47dab70fcfd591e7f6b4 [id=bd31ccb0c83f40ed80252496b582e79e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.331190 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef432f354774806adf44ed76195856f (DEFAULT_TABLE table=default.Table_2a752f83903946dea40ca65dd07107ce [id=156e2135b3ff4411bf2ebf47e6233363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.331959 5428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b6b42a01e4944dfa0f2691d4bb86446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.332912 6417 raft_consensus.cc:491] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.333315 6417 raft_consensus.cc:513] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.333652 4898 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_507494fdd4734ff1b513df2229ebddb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.335119 4898 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_507494fdd4734ff1b513df2229ebddb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.335346 6417 leader_election.cc:290] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:22.337158 5398 leader_election.cc:343] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: c813130649cc4df1b483ac4a86b4f0a0
I20250110 20:57:22.339226 5427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef432f354774806adf44ed76195856f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:22.341535 4885 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9155a626d26541ce9d0cf31508f19f15 (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9155a626d26541ce9d0cf31508f19f15 consensus state has no leader
W20250110 20:57:22.344152 4891 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 948981b34bdd4017b7ce659e7f850c5f consensus state has no leader
I20250110 20:57:22.347029 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 2ef432f354774806adf44ed76195856f (DEFAULT_TABLE table=default.Table_2a752f83903946dea40ca65dd07107ce [id=156e2135b3ff4411bf2ebf47e6233363]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.348091 5290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef432f354774806adf44ed76195856f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:22.376919 5111 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 168362us
W20250110 20:57:22.377530 5111 leader_election.cc:343] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: e508061e93cc430cb88f6d10d9684f00
I20250110 20:57:22.377853 5111 leader_election.cc:304] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:22.379745 6429 raft_consensus.cc:2743] T e508061e93cc430cb88f6d10d9684f00 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.383505 5597 tablet_bootstrap.cc:654] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:22.384950 5176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c72bb739d12244eca51224bc27f0ed1c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:22.386369 5257 leader_election.cc:343] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.386760 5257 leader_election.cc:304] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:22.386965 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c813130649cc4df1b483ac4a86b4f0a0" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:22.387751 6438 raft_consensus.cc:2743] T c72bb739d12244eca51224bc27f0ed1c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:22.387991 5399 consensus_peers.cc:487] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:22.388490 5399 leader_election.cc:343] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.388604 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:22.389187 5399 leader_election.cc:343] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.389529 5399 leader_election.cc:304] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:22.390492 5399 leader_election.cc:343] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.390802 5399 leader_election.cc:304] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:22.391662 5257 leader_election.cc:343] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.391999 5257 leader_election.cc:304] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:22.392436 6356 raft_consensus.cc:2743] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.392812 6438 raft_consensus.cc:2743] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.392890 6356 raft_consensus.cc:2743] T bae52ea080f54d228b6b76922248ed5a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.395162 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_01916dd9a30f4e8eab7f2200190902d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.396873 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01916dd9a30f4e8eab7f2200190902d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.400169 4892 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 7df3aafea18f49249cceb8a9431064e6
I20250110 20:57:22.400264 5293 tablet_service.cc:1467] Processing CreateTablet for tablet 27ee4c158df747bab2c1e97a12d86f14 (DEFAULT_TABLE table=default.Table_fce3f8bf894845e287acf83b2e2fe8c2 [id=d6e67381be374d248c8bdd8723b7a83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.401137 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ee4c158df747bab2c1e97a12d86f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.403117 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df3aafea18f49249cceb8a9431064e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:22 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:22.403662 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df3aafea18f49249cceb8a9431064e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:22 UTC) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:22.408082 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 27ee4c158df747bab2c1e97a12d86f14 (DEFAULT_TABLE table=default.Table_fce3f8bf894845e287acf83b2e2fe8c2 [id=d6e67381be374d248c8bdd8723b7a83f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:22.409080 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ee4c158df747bab2c1e97a12d86f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.411294 5301 tablet_service.cc:1514] Processing DeleteTablet for tablet 7df3aafea18f49249cceb8a9431064e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:22 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:22.411669 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a7a5b1706cc84e4e99a77621c18447ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.412986 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7a5b1706cc84e4e99a77621c18447ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:22.415273 4893 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 57) took 814 ms (client timeout 1000 ms). Trace:
I20250110 20:57:22.424006 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet e508061e93cc430cb88f6d10d9684f00 with delete_type TABLET_DATA_DELETED (replaced by tablet e324e1bdd5754864a0f65c50594c2d93) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:22.426578 5151 tablet_service.cc:1514] Processing DeleteTablet for tablet e508061e93cc430cb88f6d10d9684f00 with delete_type TABLET_DATA_DELETED (replaced by tablet e324e1bdd5754864a0f65c50594c2d93) from {username='slave'} at 127.0.0.1:35166
W20250110 20:57:22.429257 4893 rpcz_store.cc:269] 0110 20:57:21.600489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:21.604202 (+ 3713us) service_pool.cc:225] Handling call
0110 20:57:22.415222 (+811020us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:22.281782 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:22.282085 (+ 303us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:22.282105 (+ 20us) write_op.cc:435] Acquired schema lock
0110 20:57:22.282117 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:22.282392 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:22.282423 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:22.282443 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:22.282665 (+ 222us) tablet.cc:698] Row locks acquired
0110 20:57:22.282678 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:22.282765 (+ 87us) write_op.cc:270] Start()
0110 20:57:22.282832 (+ 67us) write_op.cc:276] Timestamp: P: 1736542642282683 usec, L: 0
0110 20:57:22.282846 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:22.283250 (+ 404us) log.cc:844] Serialized 913 byte log entry
0110 20:57:22.395695 (+112445us) op_driver.cc:464] REPLICATION: finished
0110 20:57:22.397367 (+ 1672us) write_op.cc:301] APPLY: starting
0110 20:57:22.397449 (+ 82us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:22.397550 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:22.397625 (+ 75us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:22.397886 (+ 261us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:22.398000 (+ 114us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:22.398038 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:57:22.398199 (+ 161us) log.cc:844] Serialized 29 byte log entry
0110 20:57:22.398310 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:22.398366 (+ 56us) write_op.cc:454] Released schema lock
0110 20:57:22.398400 (+ 34us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":1566,"apply.run_cpu_time_us":1184,"apply.run_wall_time_us":1184,"num_ops":2,"prepare.queue_time_us":21323,"prepare.run_cpu_time_us":1705,"prepare.run_wall_time_us":82268,"replication_time_us":112784}]]}
I20250110 20:57:22.445735 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27ee4c158df747bab2c1e97a12d86f14. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.447232 5141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b6b42a01e4944dfa0f2691d4bb86446. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:22.451980 5594 tablet_bootstrap.cc:654] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:22.454195 5399 consensus_peers.cc:487] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:22.455096 5399 leader_election.cc:343] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.455437 5399 leader_election.cc:304] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:22.457970 6417 raft_consensus.cc:2743] T c813130649cc4df1b483ac4a86b4f0a0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.458868 6438 raft_consensus.cc:491] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.459306 6438 raft_consensus.cc:513] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.463130 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a19079fb94aa19e2b4c85ce616f66" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:22.461063 6438 leader_election.cc:290] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:22.475025 5600 tablet_bootstrap.cc:492] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:22.475414 5600 ts_tablet_manager.cc:1397] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.366s user 0.004s sys 0.005s
I20250110 20:57:22.487879 5597 tablet_bootstrap.cc:492] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:22.488257 5597 ts_tablet_manager.cc:1397] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.426s user 0.004s sys 0.003s
W20250110 20:57:22.490373 5255 leader_election.cc:343] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.499112 4892 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_1ae0e1ebb3fa451baeb54dfb50eef3a5" } new_table_name: "default.Table_7779b0e524964a2d8e4e05e966162b39" modify_external_catalogs: true
I20250110 20:57:22.503839 4899 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_a20553538a914e1c8815acc4136592da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.506131 4899 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a20553538a914e1c8815acc4136592da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.508354 5600 raft_consensus.cc:357] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.509153 5600 raft_consensus.cc:383] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:22.509421 5600 raft_consensus.cc:738] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:22.510008 5600 consensus_queue.cc:260] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.524871 4891 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8f7a6268867c4d31b49ef33283cf5c6d" } modify_external_catalogs: true
I20250110 20:57:22.528832 5594 tablet_bootstrap.cc:492] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:22.529232 5594 ts_tablet_manager.cc:1397] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.534s user 0.014s sys 0.006s
I20250110 20:57:22.532196 5600 ts_tablet_manager.cc:1428] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.057s user 0.003s sys 0.003s
I20250110 20:57:22.532967 5600 tablet_bootstrap.cc:492] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:22.545236 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "356a19079fb94aa19e2b4c85ce616f66" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:22.546519 5257 leader_election.cc:343] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.546885 5257 leader_election.cc:304] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:22.547467 6476 raft_consensus.cc:491] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.547870 6476 raft_consensus.cc:513] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.548468 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_fce3f8bf894845e287acf83b2e2fe8c2" } new_table_name: "default.Table_5629809fdfd544aa9ea8836dbe4f9002" modify_external_catalogs: true
I20250110 20:57:22.549641 6476 leader_election.cc:290] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:22.550027 6476 raft_consensus.cc:2743] T 356a19079fb94aa19e2b4c85ce616f66 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.551172 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68d4b44975b436fb1b4a9ee6ff290d2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:22.552369 5255 leader_election.cc:343] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.554388 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68d4b44975b436fb1b4a9ee6ff290d2" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:22.554868 6454 ts_tablet_manager.cc:1918] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:22.555267 6454 log.cc:1198] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/a687769f69d844cc8564e61dc324d2b2
I20250110 20:57:22.555886 6454 ts_tablet_manager.cc:1939] T a687769f69d844cc8564e61dc324d2b2 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:22.563457 5594 raft_consensus.cc:357] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.564087 5594 raft_consensus.cc:383] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:22.564337 5594 raft_consensus.cc:738] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:22.564921 5594 consensus_queue.cc:260] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.567145 5594 ts_tablet_manager.cc:1428] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20250110 20:57:22.568048 5592 ts_tablet_manager.cc:1918] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20250110 20:57:22.568397 5592 log.cc:1198] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:22.569031 5592 ts_tablet_manager.cc:1939] T d6176c29f01b4ca5888c844c131270f2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:22.573750 6457 tablet.cc:1721] T 2644e9180a9548c08cf9e7c4bebb8189 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:22.576809 6458 tablet.cc:1721] T 2644e9180a9548c08cf9e7c4bebb8189 P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:57:22.577392 5597 raft_consensus.cc:357] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.580646 5597 raft_consensus.cc:383] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:22.581480 5597 raft_consensus.cc:738] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:22.591221 5594 tablet_bootstrap.cc:492] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:22.598259 5597 consensus_queue.cc:260] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.603740 4894 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b2e57cd89b604d4b80eadaf846bba725" } modify_external_catalogs: true
W20250110 20:57:22.612254 5111 consensus_peers.cc:487] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcc7b1acf4114bf69d801437855760df. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:22.612965 6478 raft_consensus.cc:491] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 20:57:22.613118 5257 leader_election.cc:343] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.613518 5257 leader_election.cc:304] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:22.614161 6438 raft_consensus.cc:2743] T a68d4b44975b436fb1b4a9ee6ff290d2 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.613445 6478 raft_consensus.cc:513] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.595245 5773 ts_tablet_manager.cc:1918] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:22.616318 6478 leader_election.cc:290] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:22.616819 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bedccd828042948a702fe840788765" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:22.617892 5255 leader_election.cc:343] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.630544 4896 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_6d03470d68f741a5904298647cac3f95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.631980 4896 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d03470d68f741a5904298647cac3f95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:22.629016 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 231) took 3991 ms. Trace:
I20250110 20:57:22.640645 5773 log.cc:1198] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/491626848ad54be3a18186c65d6dbb60
I20250110 20:57:22.642944 5773 ts_tablet_manager.cc:1939] T 491626848ad54be3a18186c65d6dbb60 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:22.643519 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76bedccd828042948a702fe840788765" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:22.617022 5597 ts_tablet_manager.cc:1428] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.129s user 0.006s sys 0.000s
I20250110 20:57:22.650918 5597 tablet_bootstrap.cc:492] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:22.654290 5600 tablet_bootstrap.cc:700] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384: Time spent opening tablet: real 0.117s user 0.004s sys 0.000s
I20250110 20:57:22.667305 5600 tablet_bootstrap.cc:654] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:22.668488 6480 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b84772eba3f0460faf8523f76eb57e53)
I20250110 20:57:22.641557 5592 rpcz_store.cc:276] 0110 20:57:18.636639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.636765 (+ 126us) service_pool.cc:225] Handling call
0110 20:57:22.165200 (+3528435us) ts_tablet_manager.cc:1145] Deleting tablet d6176c29f01b4ca5888c844c131270f2
0110 20:57:22.185901 (+ 20701us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.444554 (+258653us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.567847 (+123293us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.628338 (+ 60491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3527918}
I20250110 20:57:22.669200 6480 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:22.672168 6480 leader_election.cc:290] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381)
I20250110 20:57:22.672142 4833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
W20250110 20:57:22.656468 5257 leader_election.cc:343] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.679786 5257 leader_election.cc:304] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:22.680469 6478 raft_consensus.cc:2743] T 76bedccd828042948a702fe840788765 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:22.679834 5592 tablet_replica.cc:331] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:22.697768 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]) successfully deleted
I20250110 20:57:22.709527 5592 raft_consensus.cc:2238] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:22.711017 5594 tablet_bootstrap.cc:654] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:22.646888 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 237) took 4626 ms. Trace:
I20250110 20:57:22.721688 5592 raft_consensus.cc:2267] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:22.721799 5773 rpcz_store.cc:276] 0110 20:57:18.019908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.020003 (+ 95us) service_pool.cc:225] Handling call
0110 20:57:22.103394 (+4083391us) ts_tablet_manager.cc:1145] Deleting tablet 491626848ad54be3a18186c65d6dbb60
0110 20:57:22.103433 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.456778 (+353345us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.594783 (+138005us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.646591 (+ 51808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4065609}
I20250110 20:57:22.722949 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 491626848ad54be3a18186c65d6dbb60 (table default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]) successfully deleted
I20250110 20:57:22.724560 6454 tablet_replica.cc:331] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:22.724617 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet a687769f69d844cc8564e61dc324d2b2 (table default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621]) successfully deleted
I20250110 20:57:22.727207 5597 tablet_bootstrap.cc:654] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:22.736860 5600 tablet_bootstrap.cc:492] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:22.737324 5600 ts_tablet_manager.cc:1397] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.205s user 0.009s sys 0.001s
I20250110 20:57:22.739629 5600 raft_consensus.cc:357] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.740314 5600 raft_consensus.cc:383] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:22.740597 5600 raft_consensus.cc:738] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:22.741482 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:57:22.742445 4891 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_8f7a6268867c4d31b49ef33283cf5c6d" } modify_external_catalogs: true
I20250110 20:57:22.742805 6034 leader_election.cc:304] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14d1e2d5c72f498f915c367e8750361f; no voters: 63aa77ee6ec84a6e82132b2de9e77486, b84772eba3f0460faf8523f76eb57e53
I20250110 20:57:22.744925 4900 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_abcd0a50edb8458aa5f7f92aa9e0371c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:22.746412 4900 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abcd0a50edb8458aa5f7f92aa9e0371c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:22.750252 5398 consensus_peers.cc:487] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:22.743826 6480 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:57:22.760190 5592 ts_tablet_manager.cc:1905] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:22.760941 4894 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_b2e57cd89b604d4b80eadaf846bba725" } modify_external_catalogs: true
I20250110 20:57:22.774884 6205 consensus_queue.cc:1035] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:22.801846 5600 consensus_queue.cc:260] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.803793 5600 ts_tablet_manager.cc:1428] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20250110 20:57:22.811272 5594 tablet_bootstrap.cc:492] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:22.811691 5594 ts_tablet_manager.cc:1397] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.221s user 0.007s sys 0.011s
I20250110 20:57:22.815402 5592 ts_tablet_manager.cc:1918] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:22.815743 5592 log.cc:1198] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/41de354807004e26bfb147f1d420037e
I20250110 20:57:22.816360 5592 ts_tablet_manager.cc:1939] T 41de354807004e26bfb147f1d420037e P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:22.832432 4891 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet 5dd952afef9b4fde9eba5955dc0ac053
I20250110 20:57:22.833784 4891 master_service.cc:946] ReplaceTablet: received request to replace tablet 017a93be79ad49f0a72857cae4ada768 from {username='slave'} at 127.0.0.1:36568
W20250110 20:57:22.837679 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27ee4c158df747bab2c1e97a12d86f14 consensus state has no leader
I20250110 20:57:22.843927 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 240) took 4822 ms. Trace:
I20250110 20:57:22.845086 5600 tablet_bootstrap.cc:492] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:22.844915 5773 rpcz_store.cc:276] 0110 20:57:18.021034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.021253 (+ 219us) service_pool.cc:225] Handling call
0110 20:57:22.723641 (+4702388us) ts_tablet_manager.cc:1145] Deleting tablet 32cf221df30e4e78bd2a8078c26d29ba
0110 20:57:22.843587 (+119946us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.843858 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4684639}
W20250110 20:57:22.846261 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 32cf221df30e4e78bd2a8078c26d29ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32cf221df30e4e78bd2a8078c26d29ba already in progress: creating tablet
I20250110 20:57:22.846798 4895 master_service.cc:946] ReplaceTablet: received request to replace tablet f482cd146ca34f9e95b2361e9c0a985a from {username='slave'} at 127.0.0.1:36568
I20250110 20:57:22.851073 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_54bc21f0baba4cc4b8db0325939c0427 [id=8fab9a2044704cacb68b9c16330ff3ff]). Expected version 2 got 0
I20250110 20:57:22.866955 5597 tablet_bootstrap.cc:492] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:22.867437 5597 ts_tablet_manager.cc:1397] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.217s user 0.009s sys 0.004s
I20250110 20:57:22.870121 5597 raft_consensus.cc:357] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.870975 5597 raft_consensus.cc:383] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250110 20:57:22.873662 6483 log.cc:926] Time spent T 5eb75e1886e74a719dc09ccad8e1e284 P 46005a5920cc4a3697a5c61b1c271384: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20250110 20:57:22.877489 5597 raft_consensus.cc:738] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:22.878401 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 241) took 4311 ms. Trace:
I20250110 20:57:22.878306 5597 consensus_queue.cc:260] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.878692 5773 rpcz_store.cc:276] 0110 20:57:18.567335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.567441 (+ 106us) service_pool.cc:225] Handling call
0110 20:57:22.845749 (+4278308us) ts_tablet_manager.cc:1145] Deleting tablet 4b28befa048244309a06f69b8a3b4b8f
0110 20:57:22.877997 (+ 32248us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.878351 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274515}
I20250110 20:57:22.879629 6454 raft_consensus.cc:2238] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:22.880100 6454 raft_consensus.cc:2267] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:22.881840 6454 ts_tablet_manager.cc:1905] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:22.893453 5773 tablet_replica.cc:331] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:22.894112 5773 raft_consensus.cc:2238] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:22.894568 5773 raft_consensus.cc:2267] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:22.896219 5773 ts_tablet_manager.cc:1905] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:22.898950 6417 raft_consensus.cc:491] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:22.899380 6417 raft_consensus.cc:513] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.901118 6417 leader_election.cc:290] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:22.901592 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:22.901731 5597 ts_tablet_manager.cc:1428] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20250110 20:57:22.902709 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:22.902690 4893 catalog_manager.cc:5526] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:22.903775 4893 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 5eb75e1886e74a719dc09ccad8e1e284 (table default.Table_5f3f989b2ecf48a0a72e17cc18a2b095 [id=76a6826d1a6b46bf8b300259c63eb1b9]). Expected version 2 got 0
I20250110 20:57:22.904661 5594 raft_consensus.cc:357] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.905275 5594 raft_consensus.cc:383] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:22.905545 5594 raft_consensus.cc:738] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
W20250110 20:57:22.905823 5399 leader_election.cc:343] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:22.906078 5594 consensus_queue.cc:260] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:22.909082 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 241) took 3867 ms. Trace:
I20250110 20:57:22.909363 5592 rpcz_store.cc:276] 0110 20:57:19.041708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.041822 (+ 114us) service_pool.cc:225] Handling call
0110 20:57:22.675137 (+3633315us) ts_tablet_manager.cc:1145] Deleting tablet 41de354807004e26bfb147f1d420037e
0110 20:57:22.679207 (+ 4070us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.786876 (+107669us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.815204 (+ 28328us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:22.909030 (+ 93826us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9392,"tablet-delete.queue_time_us":3416437}
I20250110 20:57:22.915823 5594 ts_tablet_manager.cc:1428] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.104s user 0.006s sys 0.001s
I20250110 20:57:22.916736 6401 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: acquiring CA information for follower catalog manager: success
I20250110 20:57:22.916991 5597 tablet_bootstrap.cc:492] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:22.923563 5399 consensus_peers.cc:487] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:22.925622 5592 tablet_replica.cc:331] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:22.927220 5319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:22.927831 5319 raft_consensus.cc:2463] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
W20250110 20:57:22.887152 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 4b28befa048244309a06f69b8a3b4b8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b28befa048244309a06f69b8a3b4b8f already in progress: creating tablet
W20250110 20:57:22.933630 4887 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 46502.9us
I20250110 20:57:22.893105 4894 catalog_manager.cc:5808] Not sending DeleteTablet requests; no consensus state for tablet ac526c8a07b44e60bb795561d773f9ff
I20250110 20:57:22.875595 4891 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 017a93be79ad49f0a72857cae4ada768
I20250110 20:57:22.910670 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]) successfully deleted
I20250110 20:57:22.936038 5592 raft_consensus.cc:2238] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:22.937984 4891 catalog_manager.cc:6596] ReplaceTablet: tablet 017a93be79ad49f0a72857cae4ada768 deleted and replaced by tablet 20a087bb161a4e0f88c40790f9839e2d
I20250110 20:57:22.928922 5398 leader_election.cc:304] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:22.938251 5594 tablet_bootstrap.cc:492] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:22.938723 5592 raft_consensus.cc:2267] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:22.940485 4893 catalog_manager.cc:5526] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:22.942219 6417 raft_consensus.cc:2798] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:22.948885 6427 consensus_queue.cc:1035] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:22.950273 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet 017a93be79ad49f0a72857cae4ada768 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a087bb161a4e0f88c40790f9839e2d) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:22.951123 5594 tablet_bootstrap.cc:654] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:22.952612 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet 017a93be79ad49f0a72857cae4ada768 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a087bb161a4e0f88c40790f9839e2d) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:22.955590 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet 017a93be79ad49f0a72857cae4ada768 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a087bb161a4e0f88c40790f9839e2d) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:22.964469 5597 tablet_bootstrap.cc:654] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:22.948958 6417 raft_consensus.cc:491] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:22.967072 6417 raft_consensus.cc:3054] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:22.968314 5592 ts_tablet_manager.cc:1905] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:22.991466 4895 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet f482cd146ca34f9e95b2361e9c0a985a
I20250110 20:57:22.991672 5597 tablet_bootstrap.cc:492] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:22.992034 5597 ts_tablet_manager.cc:1397] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.001s
I20250110 20:57:22.992626 4895 catalog_manager.cc:6596] ReplaceTablet: tablet f482cd146ca34f9e95b2361e9c0a985a deleted and replaced by tablet da09f1bcbce245ccb92ddb9726ddd0c2
I20250110 20:57:22.993499 5293 tablet_service.cc:1514] Processing DeleteTablet for tablet f482cd146ca34f9e95b2361e9c0a985a with delete_type TABLET_DATA_DELETED (replaced by tablet da09f1bcbce245ccb92ddb9726ddd0c2) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:22.994241 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet f482cd146ca34f9e95b2361e9c0a985a with delete_type TABLET_DATA_DELETED (replaced by tablet da09f1bcbce245ccb92ddb9726ddd0c2) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:22.998494 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet f482cd146ca34f9e95b2361e9c0a985a with delete_type TABLET_DATA_DELETED (replaced by tablet da09f1bcbce245ccb92ddb9726ddd0c2) from {username='slave'} at 127.0.0.1:35166
I20250110 20:57:22.994709 5597 raft_consensus.cc:357] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.000228 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_0ab2fc9831784e0d8a4854264c567fd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:23.001781 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ab2fc9831784e0d8a4854264c567fd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:23.003050 5597 raft_consensus.cc:383] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.003309 5597 raft_consensus.cc:738] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.009474 5597 consensus_queue.cc:260] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.012668 5600 tablet_bootstrap.cc:654] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.020378 6417 raft_consensus.cc:513] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.021453 5597 ts_tablet_manager.cc:1428] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.029s user 0.007s sys 0.001s
I20250110 20:57:23.019133 6438 raft_consensus.cc:491] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.034087 6438 raft_consensus.cc:513] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.035185 4900 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a262069587954407a948374ed4c1b18f" } modify_external_catalogs: true
I20250110 20:57:23.035648 4900 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_a262069587954407a948374ed4c1b18f" } modify_external_catalogs: true
I20250110 20:57:23.036341 6438 leader_election.cc:290] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063)
I20250110 20:57:23.019357 6454 ts_tablet_manager.cc:1918] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:23.036988 6454 log.cc:1198] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/432078ac2fe347e1a55834ca0f77531a
I20250110 20:57:23.037783 6454 ts_tablet_manager.cc:1939] T 432078ac2fe347e1a55834ca0f77531a P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:23.042994 6427 raft_consensus.cc:491] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.043512 6427 raft_consensus.cc:513] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.045045 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0"
W20250110 20:57:23.046777 5399 leader_election.cc:343] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.038072 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484a3aebaafb4f2aa47f009b773209fe" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:23.047689 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432078ac2fe347e1a55834ca0f77531a" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:23.048055 5255 leader_election.cc:343] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:23.048481 5111 leader_election.cc:343] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 432078ac2fe347e1a55834ca0f77531a
I20250110 20:57:23.049121 5110 leader_election.cc:304] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:23.049829 6059 raft_consensus.cc:2743] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.049765 4895 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:36568:
table { table_name: "default.Table_cf6aeef756774db6a25af02ae4a44197" } new_table_name: "default.Table_39d4ef793cc541e49f867c6a36d89915" modify_external_catalogs: true
I20250110 20:57:23.027535 5773 ts_tablet_manager.cc:1918] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:23.045261 6427 leader_election.cc:290] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:23.039290 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484a3aebaafb4f2aa47f009b773209fe" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:23.052085 5257 leader_election.cc:343] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.052435 5257 leader_election.cc:304] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:23.050637 5773 log.cc:1198] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/d6176c29f01b4ca5888c844c131270f2
I20250110 20:57:23.053442 5773 ts_tablet_manager.cc:1939] T d6176c29f01b4ca5888c844c131270f2 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.053419 6438 raft_consensus.cc:2743] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.055802 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 432078ac2fe347e1a55834ca0f77531a (table default.Table_5f0518ca447a401eb8fe8655ecdc9a55 [id=ea236ef3b89c437a84ee6ec4a1cecf15]) successfully deleted
I20250110 20:57:23.062529 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 242) took 4423 ms. Trace:
I20250110 20:57:23.063046 6454 tablet_replica.cc:331] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:23.069491 6438 raft_consensus.cc:491] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.070125 6454 raft_consensus.cc:2238] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:23.069922 6438 raft_consensus.cc:513] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.070582 6454 raft_consensus.cc:2267] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.072222 6438 leader_election.cc:290] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:23.072701 5169 raft_consensus.cc:3054] T 371fb7408f0149c8b00659e09ebaf48a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.072610 5773 rpcz_store.cc:276] 0110 20:57:18.639129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:18.639251 (+ 122us) service_pool.cc:225] Handling call
0110 20:57:22.879452 (+4240201us) ts_tablet_manager.cc:1145] Deleting tablet d6176c29f01b4ca5888c844c131270f2
0110 20:57:22.893320 (+ 13868us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:22.952006 (+ 58686us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.027336 (+ 75330us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.062479 (+ 35143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4239673}
I20250110 20:57:23.073557 5170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9633eb4580c04f11a48ff5055e257c50" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:23.075606 5257 leader_election.cc:343] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.076433 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet d6176c29f01b4ca5888c844c131270f2 (table default.Table_084a1a181cf1428fb3d7af077c77c7bc [id=a4f704f781a244038555bc9f0a7ad03b]) successfully deleted
I20250110 20:57:23.078254 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9633eb4580c04f11a48ff5055e257c50" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:23.079345 5255 leader_election.cc:343] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.072289 6454 ts_tablet_manager.cc:1905] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.079669 5255 leader_election.cc:304] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:23.080250 6438 raft_consensus.cc:2743] T 9633eb4580c04f11a48ff5055e257c50 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.065907 5597 tablet_bootstrap.cc:492] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:23.084311 6464 consensus_queue.cc:1035] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:57:23.085616 4895 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79d990d874d34f98ade9b074496d5070 consensus state has no leader
I20250110 20:57:23.094079 5773 tablet_replica.cc:331] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.094811 5773 raft_consensus.cc:2238] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:23.095249 5773 raft_consensus.cc:2267] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.095973 4892 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36568:
name: "default.Table_59d177719348410c9c59e3ed9cf49ed1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:23.097048 5773 ts_tablet_manager.cc:1905] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:23.100773 4892 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59d177719348410c9c59e3ed9cf49ed1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:23.101248 4783 master-stress-test.cc:577] Killing test thread 1/10
I20250110 20:57:23.110998 6401 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 20:57:23.114715 4783 master-stress-test.cc:579] Killed test thread 1/10
I20250110 20:57:23.114953 4783 master-stress-test.cc:577] Killing test thread 2/10
I20250110 20:57:23.119665 5597 tablet_bootstrap.cc:654] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.120805 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7309d13f62524b6ab8b4c551728ecb6d" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:23.121356 5318 raft_consensus.cc:3054] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.129284 5318 raft_consensus.cc:2463] T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:23.130438 5398 leader_election.cc:304] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters: ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:23.132748 5399 consensus_peers.cc:487] T fb8c87076a8a4b1e86f7e4844e17b1f3 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:23.133289 6205 raft_consensus.cc:2798] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:23.133785 6205 raft_consensus.cc:695] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:23.134545 6205 consensus_queue.cc:237] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.142043 6417 leader_election.cc:290] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:23.145004 5773 ts_tablet_manager.cc:1918] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:23.145350 5773 log.cc:1198] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/41de354807004e26bfb147f1d420037e
I20250110 20:57:23.146845 6369 raft_consensus.cc:491] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.147249 6369 raft_consensus.cc:513] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.148578 5594 tablet_bootstrap.cc:492] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:23.148936 5594 ts_tablet_manager.cc:1397] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.211s user 0.006s sys 0.005s
I20250110 20:57:23.148967 6369 leader_election.cc:290] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:23.148976 6454 ts_tablet_manager.cc:1918] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:23.149312 6454 log.cc:1198] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/7df3aafea18f49249cceb8a9431064e6
I20250110 20:57:23.149926 6454 ts_tablet_manager.cc:1939] T 7df3aafea18f49249cceb8a9431064e6 P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:23.150527 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:23.151057 5594 raft_consensus.cc:357] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.151685 5594 raft_consensus.cc:383] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.151919 5594 raft_consensus.cc:738] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.152576 6454 tablet_replica.cc:331] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:23.152424 5594 consensus_queue.cc:260] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.152424 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 7df3aafea18f49249cceb8a9431064e6 (table default.Table_2d69d3afb9ca476cb5b329145ef8f74c [id=7016ed2e98194e4c89d508e5f724cf04]) successfully deleted
W20250110 20:57:23.156044 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 017a93be79ad49f0a72857cae4ada768 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 017a93be79ad49f0a72857cae4ada768 already in progress: creating tablet
I20250110 20:57:23.157908 5597 tablet_bootstrap.cc:492] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:23.158773 5597 ts_tablet_manager.cc:1397] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.000s
I20250110 20:57:23.158576 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2644e9180a9548c08cf9e7c4bebb8189" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:23.160442 5111 leader_election.cc:304] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac9003be04f340f3886b5625cc928eb0; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:23.161335 6427 raft_consensus.cc:2743] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.161209 5597 raft_consensus.cc:357] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.146064 5773 ts_tablet_manager.cc:1939] T 41de354807004e26bfb147f1d420037e P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.166615 5600 tablet_bootstrap.cc:492] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:23.161868 5597 raft_consensus.cc:383] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.173492 5597 raft_consensus.cc:738] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.174384 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 251) took 4162 ms. Trace:
I20250110 20:57:23.174690 5773 rpcz_store.cc:276] 0110 20:57:19.012425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.012551 (+ 126us) service_pool.cc:225] Handling call
0110 20:57:23.073542 (+4060991us) ts_tablet_manager.cc:1145] Deleting tablet 41de354807004e26bfb147f1d420037e
0110 20:57:23.093970 (+ 20428us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.128862 (+ 34892us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.144815 (+ 15953us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.174335 (+ 29520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4060465}
I20250110 20:57:23.175601 5600 ts_tablet_manager.cc:1397] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.331s user 0.006s sys 0.004s
I20250110 20:57:23.175649 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 41de354807004e26bfb147f1d420037e (table default.Table_e412e39463cf4fc0bf8a55ba265ffb25 [id=62486535b69d4a4798776d26dfdce501]) successfully deleted
I20250110 20:57:23.174175 5597 consensus_queue.cc:260] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:23.178264 6486 log.cc:926] Time spent T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Append to log took a long time: real 0.125s user 0.002s sys 0.000s
I20250110 20:57:23.180603 5597 ts_tablet_manager.cc:1428] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20250110 20:57:23.153335 6454 raft_consensus.cc:2238] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:23.184406 6454 raft_consensus.cc:2267] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.175756 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 265) took 4055 ms. Trace:
I20250110 20:57:23.190521 5773 rpcz_store.cc:276] 0110 20:57:19.120496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.197880 (+ 77384us) service_pool.cc:225] Handling call
0110 20:57:23.175523 (+3977643us) ts_tablet_manager.cc:1145] Deleting tablet f0b5ba9f38c1493bba01180744f080e1
0110 20:57:23.175559 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.175711 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3977165}
I20250110 20:57:23.191625 5600 raft_consensus.cc:357] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.192021 6500 tablet.cc:1721] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:23.193538 5597 tablet_bootstrap.cc:492] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
W20250110 20:57:23.193794 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet f0b5ba9f38c1493bba01180744f080e1 because the tablet was not found. No further retry: Not found: Tablet not found: f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:23.195140 6438 raft_consensus.cc:491] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.186241 6454 ts_tablet_manager.cc:1905] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.201634 6438 raft_consensus.cc:513] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.203349 6438 leader_election.cc:290] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:23.205315 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1330e66fc14b4da9e4a4669943360c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:23.205345 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c1330e66fc14b4da9e4a4669943360c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:23.206431 5257 leader_election.cc:343] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:23.207267 5255 leader_election.cc:343] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.207593 5255 leader_election.cc:304] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:23.208459 6438 raft_consensus.cc:2743] T 5c1330e66fc14b4da9e4a4669943360c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.211472 5600 raft_consensus.cc:383] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.217494 5600 raft_consensus.cc:738] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.221103 5600 consensus_queue.cc:260] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.222004 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 266) took 4100 ms. Trace:
I20250110 20:57:23.222298 5773 rpcz_store.cc:276] 0110 20:57:19.121113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.198399 (+ 77286us) service_pool.cc:225] Handling call
0110 20:57:23.191298 (+3992899us) ts_tablet_manager.cc:1145] Deleting tablet 8d67e94c8f9a414499b5e20a5cc0ead3
0110 20:57:23.221777 (+ 30479us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.221953 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3992483}
W20250110 20:57:23.223402 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 because the tablet was not found. No further retry: Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:23.224036 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 267) took 4098 ms. Trace:
I20250110 20:57:23.224314 5773 rpcz_store.cc:276] 0110 20:57:19.126055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.198866 (+ 72811us) service_pool.cc:225] Handling call
0110 20:57:23.223067 (+4024201us) ts_tablet_manager.cc:1145] Deleting tablet c951d5ca47e8409db646b43bb04a7034
0110 20:57:23.223835 (+ 768us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.223990 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4023735}
W20250110 20:57:23.225509 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet c951d5ca47e8409db646b43bb04a7034 because the tablet was not found. No further retry: Not found: Tablet not found: c951d5ca47e8409db646b43bb04a7034
I20250110 20:57:23.228900 4783 master-stress-test.cc:579] Killed test thread 2/10
I20250110 20:57:23.229135 4783 master-stress-test.cc:577] Killing test thread 3/10
I20250110 20:57:23.229346 4783 master-stress-test.cc:579] Killed test thread 3/10
I20250110 20:57:23.229552 4783 master-stress-test.cc:577] Killing test thread 4/10
I20250110 20:57:23.229746 4783 master-stress-test.cc:579] Killed test thread 4/10
I20250110 20:57:23.229912 4783 master-stress-test.cc:577] Killing test thread 5/10
I20250110 20:57:23.230099 4783 master-stress-test.cc:579] Killed test thread 5/10
I20250110 20:57:23.230271 4783 master-stress-test.cc:577] Killing test thread 6/10
I20250110 20:57:23.230530 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 268) took 4100 ms. Trace:
I20250110 20:57:23.230690 4783 master-stress-test.cc:579] Killed test thread 6/10
I20250110 20:57:23.230885 4783 master-stress-test.cc:577] Killing test thread 7/10
I20250110 20:57:23.231141 4783 master-stress-test.cc:579] Killed test thread 7/10
I20250110 20:57:23.230793 5773 rpcz_store.cc:276] 0110 20:57:19.130335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.199368 (+ 69033us) service_pool.cc:225] Handling call
0110 20:57:23.225113 (+4025745us) ts_tablet_manager.cc:1145] Deleting tablet 7cf0c87159574bfcbc7a34abe508f9d6
0110 20:57:23.230319 (+ 5206us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.230484 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4025333}
I20250110 20:57:23.231338 4783 master-stress-test.cc:577] Killing test thread 8/10
I20250110 20:57:23.231817 5773 tablet_replica.cc:331] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
W20250110 20:57:23.232019 4887 catalog_manager.cc:4839] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 7cf0c87159574bfcbc7a34abe508f9d6 because the tablet was not found. No further retry: Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:23.232525 5773 raft_consensus.cc:2238] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:23.233337 5773 raft_consensus.cc:2267] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.236979 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:23.237576 4898 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_54bc21f0baba4cc4b8db0325939c0427 [id=8fab9a2044704cacb68b9c16330ff3ff]). Expected version 2 got 0
I20250110 20:57:23.237949 5773 ts_tablet_manager.cc:1905] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.238473 4897 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_cdd71d3533f447558e49e1c6d6765f82 [id=9c593f8a26874512a72f0bfc20975c86] alter complete (version 2)
I20250110 20:57:23.244819 5597 tablet_bootstrap.cc:654] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.251930 5600 ts_tablet_manager.cc:1428] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.071s user 0.005s sys 0.000s
I20250110 20:57:23.254725 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet 41de354807004e26bfb147f1d420037e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:23.256935 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet f3104928188c489d93bfd499d148fa5d with delete_type TABLET_DATA_DELETED (replaced by tablet 29bbd640cb8e4adba77079181e8d1871) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:23.258695 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet c813130649cc4df1b483ac4a86b4f0a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:10 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:23.260509 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet bae52ea080f54d228b6b76922248ed5a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:23.262506 5141 tablet_service.cc:1467] Processing CreateTablet for tablet a958bc14c454436f8fe7e9fb3209c02c (DEFAULT_TABLE table=default.Table_01916dd9a30f4e8eab7f2200190902d1 [id=e3355927393e4aae904c705e8a06df0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.264045 5150 tablet_service.cc:1467] Processing CreateTablet for tablet 090e7e0f9b6d484e94009ce7512af369 (DEFAULT_TABLE table=default.Table_507494fdd4734ff1b513df2229ebddb6 [id=6dd2b0aa0f37423b9034e9bbb0a9d077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.265195 5293 tablet_service.cc:1467] Processing CreateTablet for tablet a958bc14c454436f8fe7e9fb3209c02c (DEFAULT_TABLE table=default.Table_01916dd9a30f4e8eab7f2200190902d1 [id=e3355927393e4aae904c705e8a06df0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.265861 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 090e7e0f9b6d484e94009ce7512af369 (DEFAULT_TABLE table=default.Table_507494fdd4734ff1b513df2229ebddb6 [id=6dd2b0aa0f37423b9034e9bbb0a9d077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.267197 5138 tablet_service.cc:1467] Processing CreateTablet for tablet 39ae9be8e2064965a553ae7a00a724d6 (DEFAULT_TABLE table=default.Table_a814f17e793b48a3b619651e11d38708 [id=0e8843b8fe73466dbc6d2ff520d036e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.268945 5428 tablet_service.cc:1467] Processing CreateTablet for tablet a958bc14c454436f8fe7e9fb3209c02c (DEFAULT_TABLE table=default.Table_01916dd9a30f4e8eab7f2200190902d1 [id=e3355927393e4aae904c705e8a06df0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.270102 5443 tablet_service.cc:1467] Processing CreateTablet for tablet 090e7e0f9b6d484e94009ce7512af369 (DEFAULT_TABLE table=default.Table_507494fdd4734ff1b513df2229ebddb6 [id=6dd2b0aa0f37423b9034e9bbb0a9d077]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.271281 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 39ae9be8e2064965a553ae7a00a724d6 (DEFAULT_TABLE table=default.Table_a814f17e793b48a3b619651e11d38708 [id=0e8843b8fe73466dbc6d2ff520d036e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.271715 6503 tablet.cc:1721] T 0af278622ef6432fb0891cb06bc6cd6d P ac9003be04f340f3886b5625cc928eb0: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:23.273313 5444 tablet_service.cc:1514] Processing DeleteTablet for tablet d6176c29f01b4ca5888c844c131270f2 with delete_type TABLET_DATA_DELETED (replaced by tablet 700374c672fe43c9a45f34d0255b8683) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:23.273849 5428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a958bc14c454436f8fe7e9fb3209c02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.274645 5427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ae9be8e2064965a553ae7a00a724d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.276304 5443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 090e7e0f9b6d484e94009ce7512af369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.279417 5152 tablet_service.cc:1467] Processing CreateTablet for tablet a70f93f5f0d64231afb6a16cd4abad8e (DEFAULT_TABLE table=default.Table_42c9aae4bec849858299f5a64a730abb [id=82f483968c284146b71aafddfd996879]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.280025 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 39ae9be8e2064965a553ae7a00a724d6 (DEFAULT_TABLE table=default.Table_a814f17e793b48a3b619651e11d38708 [id=0e8843b8fe73466dbc6d2ff520d036e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.280411 5152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70f93f5f0d64231afb6a16cd4abad8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.280691 5289 tablet_service.cc:1467] Processing CreateTablet for tablet a70f93f5f0d64231afb6a16cd4abad8e (DEFAULT_TABLE table=default.Table_42c9aae4bec849858299f5a64a730abb [id=82f483968c284146b71aafddfd996879]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.283813 5149 tablet_service.cc:1467] Processing CreateTablet for tablet 6969555eab21419494c854447468b6fe (DEFAULT_TABLE table=default.Table_559c738b24f74dd78374a3dbc5f1dfab [id=ec947527fc824733bcd67bf3b2bb4ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.283919 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 6969555eab21419494c854447468b6fe (DEFAULT_TABLE table=default.Table_559c738b24f74dd78374a3dbc5f1dfab [id=ec947527fc824733bcd67bf3b2bb4ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.285018 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 30bff8d59a484f878bb61937e1e5f3f8 (DEFAULT_TABLE table=default.Table_53bbb78e4859491ba4013f5dd868d449 [id=e4e3948de85c4db480e29af5b2d6c4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.285311 5151 tablet_service.cc:1467] Processing CreateTablet for tablet 30bff8d59a484f878bb61937e1e5f3f8 (DEFAULT_TABLE table=default.Table_53bbb78e4859491ba4013f5dd868d449 [id=e4e3948de85c4db480e29af5b2d6c4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.286610 6454 ts_tablet_manager.cc:1918] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:23.286962 6454 log.cc:1198] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f482cd146ca34f9e95b2361e9c0a985a
I20250110 20:57:23.287616 6454 ts_tablet_manager.cc:1939] T f482cd146ca34f9e95b2361e9c0a985a P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:23.287499 5140 tablet_service.cc:1467] Processing CreateTablet for tablet e6192c812bbe403f85cbf9b20b3fd8f4 (DEFAULT_TABLE table=default.Table_3d09af5dec5b4fcebdd16b2dcdd03e6c [id=8dc6caa6431b4999bb8996a0e2191ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.288059 5154 tablet_service.cc:1467] Processing CreateTablet for tablet 40a5b81f6b654a5d8da2acd81cdf9c07 (DEFAULT_TABLE table=default.Table_ac4a9fe3dc414f02b5e85036973f1fd2 [id=c3c2a210d2c046ae83e06768b63f0974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.289517 5300 tablet_service.cc:1467] Processing CreateTablet for tablet e6192c812bbe403f85cbf9b20b3fd8f4 (DEFAULT_TABLE table=default.Table_3d09af5dec5b4fcebdd16b2dcdd03e6c [id=8dc6caa6431b4999bb8996a0e2191ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.293530 5441 tablet_service.cc:1467] Processing CreateTablet for tablet 6969555eab21419494c854447468b6fe (DEFAULT_TABLE table=default.Table_559c738b24f74dd78374a3dbc5f1dfab [id=ec947527fc824733bcd67bf3b2bb4ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.293545 5429 tablet_service.cc:1467] Processing CreateTablet for tablet 30bff8d59a484f878bb61937e1e5f3f8 (DEFAULT_TABLE table=default.Table_53bbb78e4859491ba4013f5dd868d449 [id=e4e3948de85c4db480e29af5b2d6c4e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.295073 5440 tablet_service.cc:1467] Processing CreateTablet for tablet e6192c812bbe403f85cbf9b20b3fd8f4 (DEFAULT_TABLE table=default.Table_3d09af5dec5b4fcebdd16b2dcdd03e6c [id=8dc6caa6431b4999bb8996a0e2191ed6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.297729 5138 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ae9be8e2064965a553ae7a00a724d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.297796 5444 tablet_service.cc:1467] Processing CreateTablet for tablet a70f93f5f0d64231afb6a16cd4abad8e (DEFAULT_TABLE table=default.Table_42c9aae4bec849858299f5a64a730abb [id=82f483968c284146b71aafddfd996879]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.298592 5441 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6969555eab21419494c854447468b6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.299088 5151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bff8d59a484f878bb61937e1e5f3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.300103 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 40a5b81f6b654a5d8da2acd81cdf9c07 (DEFAULT_TABLE table=default.Table_ac4a9fe3dc414f02b5e85036973f1fd2 [id=c3c2a210d2c046ae83e06768b63f0974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.300945 5154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a5b81f6b654a5d8da2acd81cdf9c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.302011 5429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bff8d59a484f878bb61937e1e5f3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.302513 5150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 090e7e0f9b6d484e94009ce7512af369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.303582 5140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6192c812bbe403f85cbf9b20b3fd8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.305105 5149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6969555eab21419494c854447468b6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.305941 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6192c812bbe403f85cbf9b20b3fd8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.306680 5298 tablet_service.cc:1467] Processing CreateTablet for tablet 906789b19dc247379af5269c55763f5e (DEFAULT_TABLE table=default.Table_e1b3189527784d23b921008fd95b28f3 [id=6cef5b9db25b4985aab1cffcd4f33f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.307442 5141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a958bc14c454436f8fe7e9fb3209c02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.310194 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70f93f5f0d64231afb6a16cd4abad8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.311460 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 40a5b81f6b654a5d8da2acd81cdf9c07 (DEFAULT_TABLE table=default.Table_ac4a9fe3dc414f02b5e85036973f1fd2 [id=c3c2a210d2c046ae83e06768b63f0974]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.311691 5144 tablet_service.cc:1467] Processing CreateTablet for tablet 906789b19dc247379af5269c55763f5e (DEFAULT_TABLE table=default.Table_e1b3189527784d23b921008fd95b28f3 [id=6cef5b9db25b4985aab1cffcd4f33f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.312361 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a5b81f6b654a5d8da2acd81cdf9c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.312561 5144 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 906789b19dc247379af5269c55763f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.314344 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 906789b19dc247379af5269c55763f5e (DEFAULT_TABLE table=default.Table_e1b3189527784d23b921008fd95b28f3 [id=6cef5b9db25b4985aab1cffcd4f33f6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.315188 5773 ts_tablet_manager.cc:1918] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:23.315300 5426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 906789b19dc247379af5269c55763f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.315526 5773 log.cc:1198] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/c6914e258314423c859eb31e85853274
I20250110 20:57:23.316154 5773 ts_tablet_manager.cc:1939] T c6914e258314423c859eb31e85853274 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.317970 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 269) took 4186 ms. Trace:
I20250110 20:57:23.318264 5773 rpcz_store.cc:276] 0110 20:57:19.131082 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.199815 (+ 68733us) service_pool.cc:225] Handling call
0110 20:57:23.231692 (+4031877us) ts_tablet_manager.cc:1145] Deleting tablet c6914e258314423c859eb31e85853274
0110 20:57:23.231725 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.307459 (+ 75734us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.314997 (+ 7538us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.317916 (+ 2919us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4031444}
I20250110 20:57:23.321607 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet c6914e258314423c859eb31e85853274 (table default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621]) successfully deleted
I20250110 20:57:23.324395 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:23.325577 5773 tablet_replica.cc:331] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.326272 5773 raft_consensus.cc:2238] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:23.326897 5773 pending_rounds.cc:62] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Trying to abort 1 pending ops.
I20250110 20:57:23.327086 5773 pending_rounds.cc:69] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878642125389824 op_type: NO_OP noop_request { }
I20250110 20:57:23.327399 5773 raft_consensus.cc:2883] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:23.327670 5773 raft_consensus.cc:2267] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.333276 5773 ts_tablet_manager.cc:1905] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.349931 5168 raft_consensus.cc:3054] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.362080 5293 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a958bc14c454436f8fe7e9fb3209c02c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.362741 5290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 090e7e0f9b6d484e94009ce7512af369. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.363202 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 39ae9be8e2064965a553ae7a00a724d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.363699 5773 ts_tablet_manager.cc:1918] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:23.363905 5289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70f93f5f0d64231afb6a16cd4abad8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.364037 5773 log.cc:1198] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/2c1ca1670916401ab0a43685df1b6759
I20250110 20:57:23.364663 5773 ts_tablet_manager.cc:1939] T 2c1ca1670916401ab0a43685df1b6759 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.365577 5600 tablet_bootstrap.cc:492] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:23.368047 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 270) took 4234 ms. Trace:
I20250110 20:57:23.368417 5773 rpcz_store.cc:276] 0110 20:57:19.133706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.200279 (+ 66573us) service_pool.cc:225] Handling call
0110 20:57:23.325447 (+4125168us) ts_tablet_manager.cc:1145] Deleting tablet 2c1ca1670916401ab0a43685df1b6759
0110 20:57:23.325483 (+ 36us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.357492 (+ 32009us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.363521 (+ 6029us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.367996 (+ 4475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4124743}
I20250110 20:57:23.369622 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6969555eab21419494c854447468b6fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.370208 5773 tablet_replica.cc:331] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.370838 5773 raft_consensus.cc:2238] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:23.371248 5773 raft_consensus.cc:2267] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.372704 5773 ts_tablet_manager.cc:1905] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.373677 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40a5b81f6b654a5d8da2acd81cdf9c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.375969 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6192c812bbe403f85cbf9b20b3fd8f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.381641 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30bff8d59a484f878bb61937e1e5f3f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.384485 5773 ts_tablet_manager.cc:1918] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:23.384541 5600 tablet_bootstrap.cc:654] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.384786 5773 log.cc:1198] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:23.385077 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 906789b19dc247379af5269c55763f5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.385528 5773 ts_tablet_manager.cc:1939] T ba1b57de9e4b4b2ab6c3bce8fe78166a P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.387369 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 271) took 4253 ms. Trace:
I20250110 20:57:23.387627 5773 rpcz_store.cc:276] 0110 20:57:19.133970 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.200721 (+ 66751us) service_pool.cc:225] Handling call
0110 20:57:23.370066 (+4169345us) ts_tablet_manager.cc:1145] Deleting tablet ba1b57de9e4b4b2ab6c3bce8fe78166a
0110 20:57:23.370106 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.378750 (+ 8644us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.384308 (+ 5558us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.387313 (+ 3005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4168922}
I20250110 20:57:23.388792 5773 tablet_replica.cc:331] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.389432 5773 raft_consensus.cc:2238] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:23.389822 5773 raft_consensus.cc:2267] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.391386 5773 ts_tablet_manager.cc:1905] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.404299 5597 tablet_bootstrap.cc:492] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:23.404721 5597 ts_tablet_manager.cc:1397] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.211s user 0.009s sys 0.001s
I20250110 20:57:23.406874 5597 raft_consensus.cc:357] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.425565 5597 raft_consensus.cc:383] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.425900 5597 raft_consensus.cc:738] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.427599 5773 ts_tablet_manager.cc:1918] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:23.430147 6356 raft_consensus.cc:491] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 20:57:23.430256 5399 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 75607.4us
I20250110 20:57:23.430603 6356 raft_consensus.cc:513] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.432360 6356 leader_election.cc:290] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:23.434000 5398 leader_election.cc:343] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: c57941aee12a4ebea2b47f4727fc6eee
I20250110 20:57:23.441471 5773 log.cc:1198] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/da6a9a054aa54455b4979e4c633ad6b4
I20250110 20:57:23.442147 5773 ts_tablet_manager.cc:1939] T da6a9a054aa54455b4979e4c633ad6b4 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.443841 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 273) took 4081 ms. Trace:
I20250110 20:57:23.444118 5773 rpcz_store.cc:276] 0110 20:57:19.362173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.467023 (+104850us) service_pool.cc:225] Handling call
0110 20:57:23.388667 (+3921644us) ts_tablet_manager.cc:1145] Deleting tablet da6a9a054aa54455b4979e4c633ad6b4
0110 20:57:23.388701 (+ 34us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.397191 (+ 8490us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.427358 (+ 30167us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.443791 (+ 16433us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3921020}
I20250110 20:57:23.445066 5773 tablet_replica.cc:331] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.428114 5597 consensus_queue.cc:260] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.446593 5597 ts_tablet_manager.cc:1428] T c44ca47c747544b6afec83b399dcdcb5 P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.042s user 0.004s sys 0.000s
I20250110 20:57:23.447352 5597 tablet_bootstrap.cc:492] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:23.452730 5597 tablet_bootstrap.cc:654] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.460050 5773 raft_consensus.cc:2238] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:23.460878 5773 raft_consensus.cc:2267] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.462538 5773 ts_tablet_manager.cc:1905] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:23.338794 4893 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.195:33771 (request call id 58) took 821 ms (client timeout 1000 ms). Trace:
I20250110 20:57:23.369568 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 2c1ca1670916401ab0a43685df1b6759 (table default.Table_792d0af8e9d04ad28d950c7656211372 [id=176f78029038442b932e8c92372948ef]) successfully deleted
W20250110 20:57:23.467327 4887 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 97781.6us
I20250110 20:57:23.468060 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet ba1b57de9e4b4b2ab6c3bce8fe78166a (table default.Table_d5cee3d339fd4e289d63ec604aface2e [id=896c7b90b1bd4d048082ccac12d70301]) successfully deleted
I20250110 20:57:23.468662 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_853611a0967d473eab31c83128ebe9fc [id=b65f8d62ac7f44169142988e1c4903d9]) successfully deleted
I20250110 20:57:23.396281 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet f482cd146ca34f9e95b2361e9c0a985a (table default.Table_ec0889bc4a894c00bcc098b12fa053fd [id=4449f4635e6d44b6ba07a80fe9a4fa79]) successfully deleted
W20250110 20:57:23.403681 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 79d990d874d34f98ade9b074496d5070 (table default.Table_39d4ef793cc541e49f867c6a36d89915 [id=c4114f1d8e0b46b18b2c2cbd3d2430df], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 79d990d874d34f98ade9b074496d5070 consensus state has no leader
W20250110 20:57:23.470561 4886 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.194:40295, user_credentials={real_user=slave}} blocked reactor thread for 74297.6us
W20250110 20:57:23.467145 4893 rpcz_store.cc:269] 0110 20:57:22.517455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:22.518101 (+ 646us) service_pool.cc:225] Handling call
0110 20:57:23.338734 (+820633us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:22.985599 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:22.985922 (+ 323us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:22.985943 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:57:22.985956 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:22.986228 (+ 272us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:22.986307 (+ 79us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:22.986326 (+ 19us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:22.986562 (+ 236us) tablet.cc:698] Row locks acquired
0110 20:57:22.986576 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:57:22.986677 (+ 101us) write_op.cc:270] Start()
0110 20:57:22.986750 (+ 73us) write_op.cc:276] Timestamp: P: 1736542642986632 usec, L: 0
0110 20:57:22.986765 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:57:22.995681 (+ 8916us) log.cc:844] Serialized 913 byte log entry
0110 20:57:23.005712 (+ 10031us) spinlock_profiling.cc:243] Waited 8.47 ms on lock 0x7b54000316d4. stack: 00007f3bf6579853 00007f3bf6579750 00007f3bf60ee584 00000000004fcd24 00000000004fccb9 00007f3bfd4796d1 00007f3bfd4f0198 00007f3bfd95c3b1 00007f3bfd95bb88 00007f3bfd960821 00007f3bfd9607d9 00007f3bfd960769 00007f3bfd960731 00007f3bfd95f95d 000000000059ea04 000000000059e839
0110 20:57:23.042398 (+ 36686us) op_driver.cc:464] REPLICATION: finished
0110 20:57:23.043527 (+ 1129us) write_op.cc:301] APPLY: starting
0110 20:57:23.043576 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:23.043670 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:23.043687 (+ 17us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:23.043914 (+ 227us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:23.044037 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0110 20:57:23.044077 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:23.044224 (+ 147us) log.cc:844] Serialized 29 byte log entry
0110 20:57:23.044337 (+ 113us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:23.044396 (+ 59us) write_op.cc:454] Released schema lock
0110 20:57:23.044422 (+ 26us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:23.045563 (+ 1141us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:23.045586 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":19,"child_traces":[["op",{"apply.queue_time_us":980,"apply.run_cpu_time_us":1088,"apply.run_wall_time_us":1999,"num_ops":2,"prepare.queue_time_us":112,"prepare.run_cpu_time_us":2439,"prepare.run_wall_time_us":20214,"replication_time_us":55553,"spinlock_wait_cycles":19492224}]]}
I20250110 20:57:23.480401 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c57941aee12a4ebea2b47f4727fc6eee" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
I20250110 20:57:23.486936 5600 tablet_bootstrap.cc:492] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:23.487389 5600 ts_tablet_manager.cc:1397] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.122s user 0.007s sys 0.002s
I20250110 20:57:23.476243 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 5c1330e66fc14b4da9e4a4669943360c (table default.Table_03ca54bede3b459497f750a83fe9e117 [id=b78ad0741c774ec487fd064cbaaf9406]). Expected version 1 got 0
I20250110 20:57:23.488479 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 5eb75e1886e74a719dc09ccad8e1e284 (table default.Table_5f3f989b2ecf48a0a72e17cc18a2b095 [id=76a6826d1a6b46bf8b300259c63eb1b9]). Expected version 2 got 0
I20250110 20:57:23.489305 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 95b52e115a8e4bae95da0ef9d5afff8c (table default.Table_6ca3c213749745caa228b62726499ecd [id=723ca40b46694de9802850cb89dca66f]). Expected version 2 got 0
W20250110 20:57:23.491611 5399 leader_election.cc:343] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.491590 5600 raft_consensus.cc:357] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.492339 5600 raft_consensus.cc:383] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.493360 5600 raft_consensus.cc:738] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.494011 5600 consensus_queue.cc:260] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.499771 5600 ts_tablet_manager.cc:1428] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20250110 20:57:23.500610 5600 tablet_bootstrap.cc:492] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:23.504441 5597 tablet_bootstrap.cc:492] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:23.504801 5597 ts_tablet_manager.cc:1397] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.004s
I20250110 20:57:23.507334 5597 raft_consensus.cc:357] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.507947 5597 raft_consensus.cc:383] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.508185 5597 raft_consensus.cc:738] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.510654 5600 tablet_bootstrap.cc:654] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.524983 5597 consensus_queue.cc:260] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.527768 5597 ts_tablet_manager.cc:1428] T 48e53d95ef8645c0bd123138655305ac P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.023s user 0.003s sys 0.003s
I20250110 20:57:23.531615 5597 tablet_bootstrap.cc:492] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:23.491988 5399 leader_election.cc:304] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
W20250110 20:57:23.539155 5399 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 47586.5us
I20250110 20:57:23.539872 6464 raft_consensus.cc:2743] T c57941aee12a4ebea2b47f4727fc6eee P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.550899 5600 tablet_bootstrap.cc:492] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:23.551298 5600 ts_tablet_manager.cc:1397] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.017s
I20250110 20:57:23.551234 5773 ts_tablet_manager.cc:1918] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:23.551569 5773 log.cc:1198] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/5c5b954dbccb45b2a6ab7cc5cceadffb
I20250110 20:57:23.552405 5773 ts_tablet_manager.cc:1939] T 5c5b954dbccb45b2a6ab7cc5cceadffb P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.553457 5600 raft_consensus.cc:357] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.554082 5600 raft_consensus.cc:383] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.554206 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 282) took 3509 ms. Trace:
I20250110 20:57:23.554332 5600 raft_consensus.cc:738] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.554490 5773 rpcz_store.cc:276] 0110 20:57:20.045036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.109300 (+ 64264us) service_pool.cc:225] Handling call
0110 20:57:23.444938 (+3335638us) ts_tablet_manager.cc:1145] Deleting tablet 5c5b954dbccb45b2a6ab7cc5cceadffb
0110 20:57:23.444973 (+ 35us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.502212 (+ 57239us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.551000 (+ 48788us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.554153 (+ 3153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3318912}
I20250110 20:57:23.554872 5600 consensus_queue.cc:260] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.555462 5773 tablet_replica.cc:331] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.555286 5592 ts_tablet_manager.cc:1918] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:23.555863 5592 log.cc:1198] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/c6914e258314423c859eb31e85853274
I20250110 20:57:23.556095 5773 raft_consensus.cc:2238] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:23.556682 5592 ts_tablet_manager.cc:1939] T c6914e258314423c859eb31e85853274 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:23.556882 5773 raft_consensus.cc:2267] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.557773 5600 ts_tablet_manager.cc:1428] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250110 20:57:23.558542 5600 tablet_bootstrap.cc:492] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:23.558959 5597 tablet_bootstrap.cc:654] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.570148 5600 tablet_bootstrap.cc:654] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.572507 6501 tablet.cc:1721] T 0af278622ef6432fb0891cb06bc6cd6d P 46005a5920cc4a3697a5c61b1c271384: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:23.574620 5773 ts_tablet_manager.cc:1905] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.583010 5597 tablet_bootstrap.cc:492] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:23.583487 5597 ts_tablet_manager.cc:1397] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.002s
I20250110 20:57:23.584123 5600 tablet_bootstrap.cc:492] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:23.584491 5600 ts_tablet_manager.cc:1397] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.000s
I20250110 20:57:23.586105 5597 raft_consensus.cc:357] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.586827 5597 raft_consensus.cc:383] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.586555 5600 raft_consensus.cc:357] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.587137 5600 raft_consensus.cc:383] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.587150 5597 raft_consensus.cc:738] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.587379 5600 raft_consensus.cc:738] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.587862 5597 consensus_queue.cc:260] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.588099 5600 consensus_queue.cc:260] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.590097 5600 ts_tablet_manager.cc:1428] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250110 20:57:23.590907 5600 tablet_bootstrap.cc:492] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:23.596258 5600 tablet_bootstrap.cc:654] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:23.596642 5399 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 56937.2us
I20250110 20:57:23.598567 5597 ts_tablet_manager.cc:1428] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.015s user 0.001s sys 0.005s
I20250110 20:57:23.604754 5597 tablet_bootstrap.cc:492] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:23.612094 6438 raft_consensus.cc:491] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.612526 6438 raft_consensus.cc:513] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.614706 6438 leader_election.cc:290] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:23.617162 6438 raft_consensus.cc:491] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.617623 6438 raft_consensus.cc:513] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.619663 6438 leader_election.cc:290] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:23.620203 5168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efe92b10aea46608d55d5e5ac36eca1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:23.621940 5257 leader_election.cc:343] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.622679 5169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cf221df30e4e78bd2a8078c26d29ba" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac9003be04f340f3886b5625cc928eb0" is_pre_election: true
W20250110 20:57:23.623716 5257 leader_election.cc:343] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.638588 5597 tablet_bootstrap.cc:654] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.640118 5168 raft_consensus.cc:3054] T 57ef3773a87e4a23a19ac1e5f603d2ce P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.643100 5169 raft_consensus.cc:3054] T 2644e9180a9548c08cf9e7c4bebb8189 P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.528169 4900 catalog_manager.cc:5813] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: Sending DeleteTablet for 3 replicas of tablet 5bde9f66fa4647558d50e1b2febd4335
I20250110 20:57:23.556167 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 5c5b954dbccb45b2a6ab7cc5cceadffb (table default.Table_b89da7c5e86341c39853bbacce851ad4 [id=7833678313fa4c76ad1eb6d19315bb23]) successfully deleted
W20250110 20:57:23.654374 4887 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 98221.3us
I20250110 20:57:23.644048 5773 ts_tablet_manager.cc:1918] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:23.658926 5773 log.cc:1198] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/637a39e4264b40dd9480dc13e37359d9
I20250110 20:57:23.659579 5773 ts_tablet_manager.cc:1939] T 637a39e4264b40dd9480dc13e37359d9 P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.659945 5600 tablet_bootstrap.cc:492] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:23.660319 5600 ts_tablet_manager.cc:1397] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.070s user 0.044s sys 0.018s
I20250110 20:57:23.661279 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 283) took 3616 ms. Trace:
I20250110 20:57:23.661585 5773 rpcz_store.cc:276] 0110 20:57:20.045274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.126085 (+ 80811us) service_pool.cc:225] Handling call
0110 20:57:23.555335 (+3429250us) ts_tablet_manager.cc:1145] Deleting tablet 637a39e4264b40dd9480dc13e37359d9
0110 20:57:23.555372 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.631553 (+ 76181us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.643835 (+ 12282us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.661231 (+ 17396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3428683}
I20250110 20:57:23.662675 5600 raft_consensus.cc:357] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.663254 5600 raft_consensus.cc:383] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.663467 5600 raft_consensus.cc:738] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.664028 5600 consensus_queue.cc:260] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.666061 5600 ts_tablet_manager.cc:1428] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250110 20:57:23.667273 5600 tablet_bootstrap.cc:492] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:23.671804 5597 tablet_bootstrap.cc:492] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:23.672166 5597 ts_tablet_manager.cc:1397] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.000s
I20250110 20:57:23.674266 5597 raft_consensus.cc:357] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.674909 5597 raft_consensus.cc:383] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.675136 5597 raft_consensus.cc:738] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.675796 6059 raft_consensus.cc:491] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.675674 5597 consensus_queue.cc:260] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.676187 6059 raft_consensus.cc:513] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.676853 5600 tablet_bootstrap.cc:654] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.678076 6059 leader_election.cc:290] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:23.679584 5773 tablet_replica.cc:331] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.679970 5597 ts_tablet_manager.cc:1428] T 7309d13f62524b6ab8b4c551728ecb6d P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20250110 20:57:23.680402 5773 raft_consensus.cc:2238] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:23.680728 5597 tablet_bootstrap.cc:492] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:23.681167 5773 raft_consensus.cc:2267] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:23.682806 5111 leader_election.cc:343] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: f3104928188c489d93bfd499d148fa5d
W20250110 20:57:23.699967 5319 thread.cc:635] apply [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.001s sys 0.005s
W20250110 20:57:23.700333 5319 thread.cc:601] apply [worker] (thread pool) Time spent starting thread: real 0.518s user 0.001s sys 0.005s
I20250110 20:57:23.714708 5773 ts_tablet_manager.cc:1905] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.718006 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3104928188c489d93bfd499d148fa5d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:23.718587 5457 raft_consensus.cc:2463] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 0.
I20250110 20:57:23.719507 5110 leader_election.cc:304] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:23.720521 6059 raft_consensus.cc:2798] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:23.720835 6059 raft_consensus.cc:491] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:23.721109 6059 raft_consensus.cc:3054] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.721495 5594 ts_tablet_manager.cc:1428] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.572s user 0.005s sys 0.002s
I20250110 20:57:23.730947 6059 raft_consensus.cc:513] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:23.721750 5594 ts_tablet_manager.cc:1460] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6: Tablet startup took 572ms
I20250110 20:57:23.739284 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3104928188c489d93bfd499d148fa5d" candidate_uuid: "ac9003be04f340f3886b5625cc928eb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:23.739913 5457 raft_consensus.cc:3054] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:23.737237 5594 ts_tablet_manager.cc:1462] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trace:
0110 20:57:10.766219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:10.768105 (+ 1886us) service_pool.cc:225] Handling call
0110 20:57:10.769290 (+ 1185us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0110 20:57:10.769367 (+ 77us) ts_tablet_manager.cc:676] Creating new metadata...
0110 20:57:10.829699 (+ 60332us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:10.998106 (+168407us) tablet_replica.cc:206] Creating consensus instance
0110 20:57:10.998450 (+ 344us) inbound_call.cc:173] Queueing success response
0110 20:57:22.926669 (+11928219us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0110 20:57:23.149162 (+222493us) ts_tablet_manager.cc:1429] Starting tablet replica
0110 20:57:23.149687 (+ 525us) tablet_replica.cc:260] Starting instrumentation
0110 20:57:23.150112 (+ 425us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":4169,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":146,"tablet-open.queue_time_us":11928287}
I20250110 20:57:23.745915 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 257) took 4623 ms. Trace:
I20250110 20:57:23.746223 5592 rpcz_store.cc:276] 0110 20:57:19.122589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.257842 (+135253us) service_pool.cc:225] Handling call
0110 20:57:22.910278 (+3652436us) ts_tablet_manager.cc:1145] Deleting tablet c6914e258314423c859eb31e85853274
0110 20:57:22.925508 (+ 15230us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.022822 (+ 97314us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.554986 (+532164us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.745864 (+190878us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9706,"tablet-delete.queue_time_us":3651906}
I20250110 20:57:23.747354 6059 leader_election.cc:290] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:23.748119 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 258) took 4625 ms. Trace:
I20250110 20:57:23.746088 5457 raft_consensus.cc:2463] T f3104928188c489d93bfd499d148fa5d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac9003be04f340f3886b5625cc928eb0 in term 1.
W20250110 20:57:23.748127 5111 leader_election.cc:343] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: f3104928188c489d93bfd499d148fa5d
I20250110 20:57:23.751720 5110 leader_election.cc:304] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0; no voters: 46005a5920cc4a3697a5c61b1c271384
I20250110 20:57:23.752385 6427 raft_consensus.cc:2798] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:23.752790 6427 raft_consensus.cc:695] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [term 1 LEADER]: Becoming Leader. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Running, Role: LEADER
I20250110 20:57:23.753562 6427 consensus_queue.cc:237] T f3104928188c489d93bfd499d148fa5d P ac9003be04f340f3886b5625cc928eb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.759770 5459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32cf221df30e4e78bd2a8078c26d29ba" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:23.760768 5463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efe92b10aea46608d55d5e5ac36eca1" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:23.763434 5773 ts_tablet_manager.cc:1918] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:23.763757 5773 log.cc:1198] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/fcc7b1acf4114bf69d801437855760df
I20250110 20:57:23.764398 5773 ts_tablet_manager.cc:1939] T fcc7b1acf4114bf69d801437855760df P ac9003be04f340f3886b5625cc928eb0: Deleting consensus metadata
I20250110 20:57:23.750648 5597 tablet_bootstrap.cc:654] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.745265 5594 tablet_bootstrap.cc:492] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:23.778198 6520 raft_consensus.cc:491] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b84772eba3f0460faf8523f76eb57e53)
I20250110 20:57:23.778901 6520 raft_consensus.cc:513] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:23.780884 6520 leader_election.cc:290] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617), b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381)
I20250110 20:57:23.782411 4834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "63aa77ee6ec84a6e82132b2de9e77486" is_pre_election: true
I20250110 20:57:23.748405 5592 rpcz_store.cc:276] 0110 20:57:19.122887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.258414 (+135527us) service_pool.cc:225] Handling call
0110 20:57:23.747872 (+4489458us) ts_tablet_manager.cc:1145] Deleting tablet e361fb8b47d74548b5a2326d25a9e28c
0110 20:57:23.747916 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.748073 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4488968}
I20250110 20:57:23.786695 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 259) took 4663 ms. Trace:
I20250110 20:57:23.786988 5592 rpcz_store.cc:276] 0110 20:57:19.123192 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.258940 (+135748us) service_pool.cc:225] Handling call
0110 20:57:23.786425 (+4527485us) ts_tablet_manager.cc:1145] Deleting tablet 2967c06715e241dcbbe784eed2373704
0110 20:57:23.786472 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.786649 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4526996}
I20250110 20:57:23.787950 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 284) took 3742 ms. Trace:
I20250110 20:57:23.788163 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 260) took 4664 ms. Trace:
I20250110 20:57:23.788304 5597 tablet_bootstrap.cc:492] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0: No bootstrap required, opened a new log
I20250110 20:57:23.788246 5773 rpcz_store.cc:276] 0110 20:57:20.045536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.126694 (+ 81158us) service_pool.cc:225] Handling call
0110 20:57:23.662401 (+3535707us) ts_tablet_manager.cc:1145] Deleting tablet fcc7b1acf4114bf69d801437855760df
0110 20:57:23.679487 (+ 17086us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.747015 (+ 67528us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.763243 (+ 16228us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:23.787895 (+ 24652us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":109,"tablet-delete.queue_time_us":3535205}
I20250110 20:57:23.788430 5592 rpcz_store.cc:276] 0110 20:57:19.124189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.259470 (+135281us) service_pool.cc:225] Handling call
0110 20:57:23.787924 (+4528454us) ts_tablet_manager.cc:1145] Deleting tablet 2c1ca1670916401ab0a43685df1b6759
0110 20:57:23.787971 (+ 47us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.788116 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4528005}
I20250110 20:57:23.788791 5597 ts_tablet_manager.cc:1397] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.007s
I20250110 20:57:23.791026 5597 raft_consensus.cc:357] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.791682 5597 raft_consensus.cc:383] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.791921 5597 raft_consensus.cc:738] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac9003be04f340f3886b5625cc928eb0, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.795116 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 285) took 3749 ms. Trace:
I20250110 20:57:23.795423 5773 rpcz_store.cc:276] 0110 20:57:20.045804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.127244 (+ 81440us) service_pool.cc:225] Handling call
0110 20:57:23.789263 (+3662019us) ts_tablet_manager.cc:1145] Deleting tablet dd07a9eaa4364d1ca5b577a010978899
0110 20:57:23.789300 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.795066 (+ 5766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3661423}
I20250110 20:57:23.796447 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 286) took 3750 ms. Trace:
I20250110 20:57:23.796712 5773 rpcz_store.cc:276] 0110 20:57:20.046031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.127870 (+ 81839us) service_pool.cc:225] Handling call
0110 20:57:23.796165 (+3668295us) ts_tablet_manager.cc:1145] Deleting tablet 5feb54cd239b4a80894d45898776fce5
0110 20:57:23.796202 (+ 37us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.796405 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3667800}
I20250110 20:57:23.800765 5597 consensus_queue.cc:260] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:23.801443 5255 leader_election.cc:343] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.801792 5255 leader_election.cc:304] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:23.802605 6438 raft_consensus.cc:2743] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.803231 5597 ts_tablet_manager.cc:1428] T 484a3aebaafb4f2aa47f009b773209fe P ac9003be04f340f3886b5625cc928eb0: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
W20250110 20:57:23.803527 5255 leader_election.cc:343] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.803860 5255 leader_election.cc:304] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46005a5920cc4a3697a5c61b1c271384; no voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:23.804008 5597 tablet_bootstrap.cc:492] T bae52ea080f54d228b6b76922248ed5a P ac9003be04f340f3886b5625cc928eb0: Bootstrap starting.
I20250110 20:57:23.804767 6438 raft_consensus.cc:2743] T 5efe92b10aea46608d55d5e5ac36eca1 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:23.805290 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 296) took 3353 ms. Trace:
I20250110 20:57:23.805598 5773 rpcz_store.cc:276] 0110 20:57:20.451723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.585479 (+133756us) service_pool.cc:225] Handling call
0110 20:57:23.804996 (+3219517us) ts_tablet_manager.cc:1145] Deleting tablet 0e9c229f569c4468bf43e7e880f7f0df
0110 20:57:23.805037 (+ 41us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.805246 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3211398}
I20250110 20:57:23.806582 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 295) took 3355 ms. Trace:
I20250110 20:57:23.806833 5773 rpcz_store.cc:276] 0110 20:57:20.451471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:20.490800 (+ 39329us) service_pool.cc:225] Handling call
0110 20:57:23.806323 (+3315523us) ts_tablet_manager.cc:1145] Deleting tablet e324e1bdd5754864a0f65c50594c2d93
0110 20:57:23.806355 (+ 32us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.806542 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3079428}
I20250110 20:57:23.807778 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 301) took 1925 ms. Trace:
I20250110 20:57:23.808027 5773 rpcz_store.cc:276] 0110 20:57:21.882697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:21.884121 (+ 1424us) service_pool.cc:225] Handling call
0110 20:57:23.807520 (+1923399us) ts_tablet_manager.cc:1145] Deleting tablet adfed905cadb4e82acd3ff066c8c4cba
0110 20:57:23.807553 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.807739 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1897923}
I20250110 20:57:23.809015 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 302) took 1814 ms. Trace:
I20250110 20:57:23.815491 5597 tablet_bootstrap.cc:654] T bae52ea080f54d228b6b76922248ed5a P ac9003be04f340f3886b5625cc928eb0: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.818956 5600 tablet_bootstrap.cc:492] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:23.819339 5600 ts_tablet_manager.cc:1397] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.152s user 0.014s sys 0.013s
I20250110 20:57:23.821358 5600 raft_consensus.cc:357] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.822002 5600 raft_consensus.cc:383] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:23.822252 5600 raft_consensus.cc:738] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:23.822964 5600 consensus_queue.cc:260] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:23.510285 4898 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 2644e9180a9548c08cf9e7c4bebb8189 (table default.Table_54bc21f0baba4cc4b8db0325939c0427 [id=8fab9a2044704cacb68b9c16330ff3ff]). Expected version 2 got 0
I20250110 20:57:23.824997 4898 catalog_manager.cc:5618] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:23.825708 4897 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_54bc21f0baba4cc4b8db0325939c0427 [id=8fab9a2044704cacb68b9c16330ff3ff] alter complete (version 2)
I20250110 20:57:23.809304 5773 rpcz_store.cc:276] 0110 20:57:21.994803 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:21.994932 (+ 129us) service_pool.cc:225] Handling call
0110 20:57:23.808752 (+1813820us) ts_tablet_manager.cc:1145] Deleting tablet 51fae5f59d4642a09acee5a687b91717
0110 20:57:23.808785 (+ 33us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.808977 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813305}
I20250110 20:57:23.828274 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet 637a39e4264b40dd9480dc13e37359d9 (table default.Table_638a8034373e4e3bbfb842beaacdaab8 [id=42aa140491c94323b9a140eea45f28de]) successfully deleted
I20250110 20:57:23.831467 5600 ts_tablet_manager.cc:1428] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20250110 20:57:23.832156 5290 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bde9f66fa4647558d50e1b2febd4335 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:23 UTC) from {username='slave'} at 127.0.0.1:55028
W20250110 20:57:23.833638 4886 catalog_manager.cc:4852] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): delete failed for tablet 5bde9f66fa4647558d50e1b2febd4335 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bde9f66fa4647558d50e1b2febd4335 already in progress: creating tablet
I20250110 20:57:23.838817 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 261) took 4714 ms. Trace:
I20250110 20:57:23.838779 4887 catalog_manager.cc:4872] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): tablet fcc7b1acf4114bf69d801437855760df (table default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]) successfully deleted
I20250110 20:57:23.839278 5592 rpcz_store.cc:276] 0110 20:57:19.124461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.259963 (+135502us) service_pool.cc:225] Handling call
0110 20:57:23.838538 (+4578575us) ts_tablet_manager.cc:1145] Deleting tablet 7cf0c87159574bfcbc7a34abe508f9d6
0110 20:57:23.838591 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.838765 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4578112}
I20250110 20:57:23.840670 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 262) took 4716 ms. Trace:
I20250110 20:57:23.841118 5592 rpcz_store.cc:276] 0110 20:57:19.124740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.260464 (+135724us) service_pool.cc:225] Handling call
0110 20:57:23.840404 (+4579940us) ts_tablet_manager.cc:1145] Deleting tablet 527bbddda8184a6eb67c2411ae50db63
0110 20:57:23.840449 (+ 45us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.840627 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4579475}
I20250110 20:57:23.845659 5773 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35166 (request call id 303) took 1586 ms. Trace:
I20250110 20:57:23.845956 5773 rpcz_store.cc:276] 0110 20:57:22.259201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:22.267725 (+ 8524us) service_pool.cc:225] Handling call
0110 20:57:23.845396 (+1577671us) ts_tablet_manager.cc:1145] Deleting tablet 527bbddda8184a6eb67c2411ae50db63
0110 20:57:23.845456 (+ 60us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.845604 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1558586}
I20250110 20:57:23.847090 5773 tablet_replica.cc:331] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0: stopping tablet replica
I20250110 20:57:23.847007 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet c6914e258314423c859eb31e85853274 (table default.Table_a73e1fb622794aa4a8a231555b3d53ca [id=a02df045e79245db9b8b7ad158771621]) successfully deleted
W20250110 20:57:23.847589 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet e361fb8b47d74548b5a2326d25a9e28c because the tablet was not found. No further retry: Not found: Tablet not found: e361fb8b47d74548b5a2326d25a9e28c
I20250110 20:57:23.847786 5773 raft_consensus.cc:2238] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:23.848229 5773 raft_consensus.cc:2267] T 432078ac2fe347e1a55834ca0f77531a P ac9003be04f340f3886b5625cc928eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.849221 4783 master-stress-test.cc:579] Killed test thread 8/10
I20250110 20:57:23.849490 4783 master-stress-test.cc:577] Killing test thread 9/10
I20250110 20:57:23.849757 4783 master-stress-test.cc:579] Killed test thread 9/10
I20250110 20:57:23.849934 4783 master-stress-test.cc:577] Killing test thread 10/10
I20250110 20:57:23.850176 4783 master-stress-test.cc:579] Killed test thread 10/10
I20250110 20:57:23.850495 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 5062
I20250110 20:57:23.851783 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "14d1e2d5c72f498f915c367e8750361f" candidate_term: 3 candidate_status { last_received { term: 2 index: 261 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:57:23.853067 6034 leader_election.cc:304] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 14d1e2d5c72f498f915c367e8750361f; no voters: 63aa77ee6ec84a6e82132b2de9e77486, b84772eba3f0460faf8523f76eb57e53
W20250110 20:57:23.857643 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet dd07a9eaa4364d1ca5b577a010978899 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dd07a9eaa4364d1ca5b577a010978899 already in progress: creating tablet
W20250110 20:57:23.858269 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 5feb54cd239b4a80894d45898776fce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5feb54cd239b4a80894d45898776fce5 already in progress: creating tablet
W20250110 20:57:23.858829 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet 0e9c229f569c4468bf43e7e880f7f0df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e9c229f569c4468bf43e7e880f7f0df already in progress: creating tablet
W20250110 20:57:23.859354 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet e324e1bdd5754864a0f65c50594c2d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e324e1bdd5754864a0f65c50594c2d93 already in progress: creating tablet
W20250110 20:57:23.859846 4887 catalog_manager.cc:4852] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): delete failed for tablet adfed905cadb4e82acd3ff066c8c4cba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet adfed905cadb4e82acd3ff066c8c4cba already in progress: creating tablet
I20250110 20:57:23.865545 6520 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 14d1e2d5c72f498f915c367e8750361f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250110 20:57:23.848124 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 2967c06715e241dcbbe784eed2373704 because the tablet was not found. No further retry: Not found: Tablet not found: 2967c06715e241dcbbe784eed2373704
W20250110 20:57:23.867790 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 2c1ca1670916401ab0a43685df1b6759 because the tablet was not found. No further retry: Not found: Tablet not found: 2c1ca1670916401ab0a43685df1b6759
W20250110 20:57:23.868367 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 7cf0c87159574bfcbc7a34abe508f9d6 because the tablet was not found. No further retry: Not found: Tablet not found: 7cf0c87159574bfcbc7a34abe508f9d6
I20250110 20:57:23.867012 5600 tablet_bootstrap.cc:492] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:23.876520 4885 catalog_manager.cc:4945] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): alter failed for tablet d310cf52aa8c4930a7821117a46dc958 (table default.Table_5053ea7526a74f6da2ece254f837bbae [id=041d33bef58a41529a7738d9e406af1f]),no further retry: Not found: Tablet not found: d310cf52aa8c4930a7821117a46dc958
I20250110 20:57:23.897724 5296 tablet_service.cc:1467] Processing CreateTablet for tablet 480574a892a3432c8c3b1e2561add0e1 (DEFAULT_TABLE table=default.Table_0ab2fc9831784e0d8a4854264c567fd1 [id=8a970422a13147edbc154dfb7c8188eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.898643 5296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480574a892a3432c8c3b1e2561add0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.900226 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 256) took 4781 ms. Trace:
I20250110 20:57:23.900556 5592 rpcz_store.cc:276] 0110 20:57:19.118972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.119135 (+ 163us) service_pool.cc:225] Handling call
0110 20:57:23.899969 (+4780834us) ts_tablet_manager.cc:1145] Deleting tablet ba1b57de9e4b4b2ab6c3bce8fe78166a
0110 20:57:23.900019 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:23.900180 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4635812}
I20250110 20:57:23.901516 5592 tablet_replica.cc:331] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:23.876727 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bde9f66fa4647558d50e1b2febd4335 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:23 UTC) from {username='slave'} at 127.0.0.1:40232
W20250110 20:57:23.901510 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 527bbddda8184a6eb67c2411ae50db63 because the tablet was not found. No further retry: Not found: Tablet not found: 527bbddda8184a6eb67c2411ae50db63
W20250110 20:57:23.902164 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet ba1b57de9e4b4b2ab6c3bce8fe78166a because the tablet was not found. No further retry: Not found: Tablet not found: ba1b57de9e4b4b2ab6c3bce8fe78166a
I20250110 20:57:23.890730 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9ba67c845547e7a1d902791241015c (DEFAULT_TABLE table=default.Table_59d177719348410c9c59e3ed9cf49ed1 [id=da3367d1df6e4a8b977fefecf384ce6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.903091 5427 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9ba67c845547e7a1d902791241015c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.892413 5426 tablet_service.cc:1467] Processing CreateTablet for tablet b07c9f7b09044704bf7e06250e1cefea (DEFAULT_TABLE table=default.Table_abcd0a50edb8458aa5f7f92aa9e0371c [id=5cf221d6a8274d3092078b5e89ef0cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.898494 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 480574a892a3432c8c3b1e2561add0e1 (DEFAULT_TABLE table=default.Table_0ab2fc9831784e0d8a4854264c567fd1 [id=8a970422a13147edbc154dfb7c8188eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.910108 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 480574a892a3432c8c3b1e2561add0e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.899063 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 92083f7246a242a6bb75cefd38436c73 (DEFAULT_TABLE table=default.Table_a20553538a914e1c8815acc4136592da [id=d320bd16c78e47e9bc6890df50a61cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.914525 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92083f7246a242a6bb75cefd38436c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.843914 5594 tablet_bootstrap.cc:654] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.921540 6417 raft_consensus.cc:491] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.921959 6417 raft_consensus.cc:513] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.902220 5592 raft_consensus.cc:2238] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:23.924965 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 8c9ba67c845547e7a1d902791241015c (DEFAULT_TABLE table=default.Table_59d177719348410c9c59e3ed9cf49ed1 [id=da3367d1df6e4a8b977fefecf384ce6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.926290 5592 raft_consensus.cc:2267] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:23.927909 5290 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c9ba67c845547e7a1d902791241015c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.929580 5600 tablet_bootstrap.cc:654] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:23.933501 6417 leader_election.cc:290] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:23.934571 5426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07c9f7b09044704bf7e06250e1cefea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.939172 5298 tablet_service.cc:1467] Processing CreateTablet for tablet b07c9f7b09044704bf7e06250e1cefea (DEFAULT_TABLE table=default.Table_abcd0a50edb8458aa5f7f92aa9e0371c [id=5cf221d6a8274d3092078b5e89ef0cac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.940184 5298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b07c9f7b09044704bf7e06250e1cefea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.902777 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:23.941293 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484a3aebaafb4f2aa47f009b773209fe" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:23.941865 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 5eb75e1886e74a719dc09ccad8e1e284 (table default.Table_5f3f989b2ecf48a0a72e17cc18a2b095 [id=76a6826d1a6b46bf8b300259c63eb1b9]). Expected version 2 got 0
I20250110 20:57:23.941888 5322 raft_consensus.cc:2463] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:23.942763 4900 catalog_manager.cc:5526] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:23.943893 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_5675215f702a41ab934b6baf97fd0431 [id=1c2b9e8ff5d341c380b50023c577d0aa]). Expected version 2 got 0
I20250110 20:57:23.924611 6438 raft_consensus.cc:491] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:23.946691 6438 raft_consensus.cc:513] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.948501 6438 leader_election.cc:290] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:23.949962 5398 leader_election.cc:304] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:23.951187 6356 raft_consensus.cc:2798] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:23.951501 6356 raft_consensus.cc:491] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:23.951788 6356 raft_consensus.cc:3054] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.952163 6205 consensus_queue.cc:1035] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:23.956630 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b52e115a8e4bae95da0ef9d5afff8c" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:23.958922 6356 raft_consensus.cc:513] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:23.961009 6356 leader_election.cc:290] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:23.961900 5255 leader_election.cc:343] T 95b52e115a8e4bae95da0ef9d5afff8c P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:23.964319 5322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "484a3aebaafb4f2aa47f009b773209fe" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:23.964848 5322 raft_consensus.cc:3054] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:23.975312 5303 tablet_service.cc:1467] Processing CreateTablet for tablet 92083f7246a242a6bb75cefd38436c73 (DEFAULT_TABLE table=default.Table_a20553538a914e1c8815acc4136592da [id=d320bd16c78e47e9bc6890df50a61cf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.976275 5303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92083f7246a242a6bb75cefd38436c73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.979164 5297 tablet_service.cc:1467] Processing CreateTablet for tablet 837df26eb04d491383c54b7211796858 (DEFAULT_TABLE table=default.Table_a7a5b1706cc84e4e99a77621c18447ec [id=5d3db8fba7984f22b33b5af96bdcb1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.980055 5297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 837df26eb04d491383c54b7211796858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.981875 5300 tablet_service.cc:1467] Processing CreateTablet for tablet 20a087bb161a4e0f88c40790f9839e2d (DEFAULT_TABLE table=default.Table_36d9c90774f84119abe713d7b6331a49 [id=1441f7755521427390ee18c201f2421b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.982717 5300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a087bb161a4e0f88c40790f9839e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.986207 5444 tablet_service.cc:1467] Processing CreateTablet for tablet 837df26eb04d491383c54b7211796858 (DEFAULT_TABLE table=default.Table_a7a5b1706cc84e4e99a77621c18447ec [id=5d3db8fba7984f22b33b5af96bdcb1ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:23.987107 5444 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 837df26eb04d491383c54b7211796858. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:23.990484 5442 tablet_service.cc:1467] Processing CreateTablet for tablet da09f1bcbce245ccb92ddb9726ddd0c2 (DEFAULT_TABLE table=default.Table_ec0889bc4a894c00bcc098b12fa053fd [id=4449f4635e6d44b6ba07a80fe9a4fa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:24.002470 5301 tablet_service.cc:1467] Processing CreateTablet for tablet 856b38fc5dc7449d805db2d68111bd74 (DEFAULT_TABLE table=default.Table_6d03470d68f741a5904298647cac3f95 [id=20f2ee4121da465e8085bf52d21071d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:24.003365 5301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856b38fc5dc7449d805db2d68111bd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:24.014142 5289 tablet_service.cc:1467] Processing CreateTablet for tablet da09f1bcbce245ccb92ddb9726ddd0c2 (DEFAULT_TABLE table=default.Table_ec0889bc4a894c00bcc098b12fa053fd [id=4449f4635e6d44b6ba07a80fe9a4fa79]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:24.014997 5594 tablet_bootstrap.cc:492] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.015094 5289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da09f1bcbce245ccb92ddb9726ddd0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:24.015377 5594 ts_tablet_manager.cc:1397] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.270s user 0.007s sys 0.001s
I20250110 20:57:24.021996 5594 raft_consensus.cc:357] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.022745 5594 raft_consensus.cc:383] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.022671 5428 tablet_service.cc:1467] Processing CreateTablet for tablet 20a087bb161a4e0f88c40790f9839e2d (DEFAULT_TABLE table=default.Table_36d9c90774f84119abe713d7b6331a49 [id=1441f7755521427390ee18c201f2421b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:24.023051 5594 raft_consensus.cc:738] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.018003 5440 tablet_service.cc:1467] Processing CreateTablet for tablet 856b38fc5dc7449d805db2d68111bd74 (DEFAULT_TABLE table=default.Table_6d03470d68f741a5904298647cac3f95 [id=20f2ee4121da465e8085bf52d21071d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:24.023661 5594 consensus_queue.cc:260] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.025447 5592 ts_tablet_manager.cc:1905] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:23.975694 5322 raft_consensus.cc:2463] T 484a3aebaafb4f2aa47f009b773209fe P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:24.034910 6305 mvcc.cc:204] Tried to move back new op lower bound from 7112878636103225344 to 7112878578867171328. Current Snapshot: MvccSnapshot[applied={T|T < 7112878636103225344 or (T in {7112878636103225344})}]
I20250110 20:57:24.038195 5442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da09f1bcbce245ccb92ddb9726ddd0c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:24.041544 5398 leader_election.cc:304] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:24.042230 5594 ts_tablet_manager.cc:1428] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20250110 20:57:24.044700 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 95b52e115a8e4bae95da0ef9d5afff8c (table default.Table_6ca3c213749745caa228b62726499ecd [id=723ca40b46694de9802850cb89dca66f]). Expected version 2 got 0
I20250110 20:57:24.045974 6417 raft_consensus.cc:2798] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:24.046005 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet bb535174548d496e920801723e0909b3 (table default.Table_eeeff6d7c2e9444bb129b22fe16b8f1c [id=83c23c863af4460296277c44bddc35fb]). Expected version 2 got 0
I20250110 20:57:24.046402 6417 raft_consensus.cc:695] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:24.046713 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 79d990d874d34f98ade9b074496d5070 (table default.Table_39d4ef793cc541e49f867c6a36d89915 [id=c4114f1d8e0b46b18b2c2cbd3d2430df]). Expected version 1 got 0
I20250110 20:57:24.047083 6417 consensus_queue.cc:237] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.054314 4900 catalog_manager.cc:6071] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53: default.Table_7779b0e524964a2d8e4e05e966162b39 [id=0df25099e0f4482a8881a258a954312a] alter complete (version 1)
I20250110 20:57:24.060146 5441 tablet_service.cc:1514] Processing DeleteTablet for tablet c57941aee12a4ebea2b47f4727fc6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:15 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:24.077739 5594 tablet_bootstrap.cc:492] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:24.093081 5428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a087bb161a4e0f88c40790f9839e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:24.101662 5440 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 856b38fc5dc7449d805db2d68111bd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:24.112017 5594 tablet_bootstrap.cc:654] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.152637 4897 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 371fb7408f0149c8b00659e09ebaf48a (table default.Table_057bb914f63b44f28527df30e38ce58f [id=c40af9e5c0e5472580ecfe5423b124e5]). Expected version 3 got 0
I20250110 20:57:24.153385 4897 catalog_manager.cc:5526] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:24.164023 5592 ts_tablet_manager.cc:1918] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:24.164425 5592 log.cc:1198] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/491626848ad54be3a18186c65d6dbb60
I20250110 20:57:24.165078 5592 ts_tablet_manager.cc:1939] T 491626848ad54be3a18186c65d6dbb60 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:24.167857 5297 tablet_service.cc:1514] Processing DeleteTablet for tablet 32cf221df30e4e78bd2a8078c26d29ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:24.169333 6454 tablet_replica.cc:331] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:24.170279 6454 raft_consensus.cc:2238] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:24.170831 6454 raft_consensus.cc:2267] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:24.172667 6454 ts_tablet_manager.cc:1905] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:24.182960 5594 tablet_bootstrap.cc:492] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.183363 5594 ts_tablet_manager.cc:1397] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.106s user 0.006s sys 0.002s
I20250110 20:57:24.185638 5594 raft_consensus.cc:357] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.186267 5594 raft_consensus.cc:383] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.186507 5594 raft_consensus.cc:738] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.187047 5594 consensus_queue.cc:260] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
W20250110 20:57:24.190057 6527 log.cc:926] Time spent T 7309d13f62524b6ab8b4c551728ecb6d P 46005a5920cc4a3697a5c61b1c271384: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20250110 20:57:24.202244 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_5675215f702a41ab934b6baf97fd0431 [id=1c2b9e8ff5d341c380b50023c577d0aa]). Expected version 2 got 0
I20250110 20:57:24.217629 6529 raft_consensus.cc:491] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:24.218063 6529 raft_consensus.cc:513] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.219797 6529 leader_election.cc:290] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:24.220232 6529 raft_consensus.cc:491] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:24.220575 6529 raft_consensus.cc:513] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.242259 6529 leader_election.cc:290] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:24.254557 5289 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b28befa048244309a06f69b8a3b4b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:18 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:24.254976 5600 tablet_bootstrap.cc:492] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:24.255374 5600 ts_tablet_manager.cc:1397] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.394s user 0.013s sys 0.043s
I20250110 20:57:24.255792 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb535174548d496e920801723e0909b3" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:24.257570 5600 raft_consensus.cc:357] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.258175 5600 raft_consensus.cc:383] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.258091 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b28befa048244309a06f69b8a3b4b8f" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:24.258421 5600 raft_consensus.cc:738] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.258924 5600 consensus_queue.cc:260] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
W20250110 20:57:24.260224 5255 leader_election.cc:343] T bb535174548d496e920801723e0909b3 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:24.260761 5255 leader_election.cc:343] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:24.260866 5600 ts_tablet_manager.cc:1428] T 62fe5d3bd809412a8bce721a57406f84 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250110 20:57:24.261754 5600 tablet_bootstrap.cc:492] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:24.275219 5600 tablet_bootstrap.cc:654] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.293449 4897 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_5675215f702a41ab934b6baf97fd0431 [id=1c2b9e8ff5d341c380b50023c577d0aa]). Expected version 2 got 0
I20250110 20:57:24.303153 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 264) took 5174 ms. Trace:
I20250110 20:57:24.303730 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 62fe5d3bd809412a8bce721a57406f84 (table default.Table_76e5da9e2e7e4e58a06bc9d08ffbd8c2 [id=82b804310c184b63bbc27b249f912283]). Expected version 1 got 0
I20250110 20:57:24.321316 5600 tablet_bootstrap.cc:492] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:24.321734 5600 ts_tablet_manager.cc:1397] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.002s
I20250110 20:57:24.324012 5600 raft_consensus.cc:357] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.324636 5600 raft_consensus.cc:383] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.324873 5600 raft_consensus.cc:738] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.327996 5600 consensus_queue.cc:260] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.330005 5600 ts_tablet_manager.cc:1428] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250110 20:57:24.330214 6454 ts_tablet_manager.cc:1918] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:24.330792 5600 tablet_bootstrap.cc:492] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:24.335829 5600 tablet_bootstrap.cc:654] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.342001 4900 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 62fe5d3bd809412a8bce721a57406f84 (table default.Table_76e5da9e2e7e4e58a06bc9d08ffbd8c2 [id=82b804310c184b63bbc27b249f912283]). Expected version 1 got 0
I20250110 20:57:24.344389 6532 raft_consensus.cc:491] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:24.344843 6532 raft_consensus.cc:513] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.347771 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79d990d874d34f98ade9b074496d5070" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
W20250110 20:57:24.348862 5255 leader_election.cc:343] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:24.349740 6532 leader_election.cc:290] T 79d990d874d34f98ade9b074496d5070 P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:24.350558 5594 ts_tablet_manager.cc:1428] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.167s user 0.006s sys 0.000s
I20250110 20:57:24.303473 5592 rpcz_store.cc:276] 0110 20:57:19.128821 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.264206 (+135385us) service_pool.cc:225] Handling call
0110 20:57:23.901345 (+4637139us) ts_tablet_manager.cc:1145] Deleting tablet 491626848ad54be3a18186c65d6dbb60
0110 20:57:23.901397 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.103737 (+202340us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.163811 (+ 60074us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.303097 (+139286us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4636652}
I20250110 20:57:24.353869 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 491626848ad54be3a18186c65d6dbb60 (table default.Table_8e63a6ded00f470d94804f6f15343b29 [id=59dd5a3854f546c2b6ff15880c73616a]) successfully deleted
I20250110 20:57:24.356158 5594 tablet_bootstrap.cc:492] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:24.360673 4897 catalog_manager.cc:5618] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295) does not have the latest schema for tablet 7309d13f62524b6ab8b4c551728ecb6d (table default.Table_5675215f702a41ab934b6baf97fd0431 [id=1c2b9e8ff5d341c380b50023c577d0aa]). Expected version 2 got 0
I20250110 20:57:24.361048 5594 tablet_bootstrap.cc:654] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.365736 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 265) took 5236 ms. Trace:
I20250110 20:57:24.366039 5592 rpcz_store.cc:276] 0110 20:57:19.129072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.264739 (+135667us) service_pool.cc:225] Handling call
0110 20:57:24.351122 (+5086383us) ts_tablet_manager.cc:1145] Deleting tablet 8d67e94c8f9a414499b5e20a5cc0ead3
0110 20:57:24.365488 (+ 14366us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.365685 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5085871}
I20250110 20:57:24.366933 5592 tablet_replica.cc:331] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:24.367590 5592 raft_consensus.cc:2238] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250110 20:57:24.367580 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 8d67e94c8f9a414499b5e20a5cc0ead3 because the tablet was not found. No further retry: Not found: Tablet not found: 8d67e94c8f9a414499b5e20a5cc0ead3
I20250110 20:57:24.368006 5592 raft_consensus.cc:2267] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:24.368896 5289 tablet_service.cc:1514] Processing DeleteTablet for tablet a95f05fc5afe4877a606b501dbd09551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:16 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:24.330579 6454 log.cc:1198] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/32cf221df30e4e78bd2a8078c26d29ba
I20250110 20:57:24.382000 6454 ts_tablet_manager.cc:1939] T 32cf221df30e4e78bd2a8078c26d29ba P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:24.384446 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 32cf221df30e4e78bd2a8078c26d29ba (table default.Table_3f8bafa0c19642adb6062c8156965b27 [id=f0fdb4ee34df445b9b8cf660d3ae239d]) successfully deleted
I20250110 20:57:24.385248 6454 tablet_replica.cc:331] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:24.405619 5592 ts_tablet_manager.cc:1905] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:24.441792 6454 raft_consensus.cc:2238] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:24.442297 6454 raft_consensus.cc:2267] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:24.444036 6454 ts_tablet_manager.cc:1905] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:24.475553 5600 tablet_bootstrap.cc:492] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:24.475945 5600 ts_tablet_manager.cc:1397] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.145s user 0.010s sys 0.018s
I20250110 20:57:24.477988 5600 raft_consensus.cc:357] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.478597 5600 raft_consensus.cc:383] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.478816 5600 raft_consensus.cc:738] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.480120 5600 consensus_queue.cc:260] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.482563 5600 ts_tablet_manager.cc:1428] T 16aaabfff9a04310b19a3195d0b41d7a P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20250110 20:57:20.834120 6121 rpcz_store.cc:276] 0110 20:57:16.516360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:16.516605 (+ 245us) service_pool.cc:225] Handling call
0110 20:57:18.532113 (+2015508us) raft_consensus.cc:1466] Updating replica for 252 ops
0110 20:57:18.682083 (+149970us) raft_consensus.cc:1529] Early marking committed up to index 9
0110 20:57:18.723675 (+ 41592us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:18.723700 (+ 25us) write_op.cc:454] Released schema lock
0110 20:57:18.723863 (+ 163us) raft_consensus.cc:1534] Triggering prepare for 252 ops
0110 20:57:18.724427 (+ 564us) write_op.cc:270] Start()
0110 20:57:18.724523 (+ 96us) write_op.cc:276] Timestamp: P: 1736542622528993 usec, L: 0
0110 20:57:18.739625 (+ 15102us) write_op.cc:270] Start()
0110 20:57:18.741937 (+ 2312us) write_op.cc:276] Timestamp: P: 1736542622544297 usec, L: 0
0110 20:57:18.743888 (+ 1951us) write_op.cc:270] Start()
0110 20:57:18.743975 (+ 87us) write_op.cc:276] Timestamp: P: 1736542622562665 usec, L: 0
0110 20:57:18.745973 (+ 1998us) write_op.cc:270] Start()
0110 20:57:18.746085 (+ 112us) write_op.cc:276] Timestamp: P: 1736542622576433 usec, L: 0
0110 20:57:18.755492 (+ 9407us) write_op.cc:270] Start()
0110 20:57:18.755579 (+ 87us) write_op.cc:276] Timestamp: P: 1736542622578378 usec, L: 0
0110 20:57:18.760372 (+ 4793us) write_op.cc:270] Start()
0110 20:57:18.760469 (+ 97us) write_op.cc:276] Timestamp: P: 1736542622589699 usec, L: 0
0110 20:57:18.763141 (+ 2672us) write_op.cc:270] Start()
0110 20:57:18.763215 (+ 74us) write_op.cc:276] Timestamp: P: 1736542622592930 usec, L: 0
0110 20:57:18.770709 (+ 7494us) write_op.cc:270] Start()
0110 20:57:18.770792 (+ 83us) write_op.cc:276] Timestamp: P: 1736542622615383 usec, L: 0
0110 20:57:18.773320 (+ 2528us) write_op.cc:270] Start()
0110 20:57:18.773398 (+ 78us) write_op.cc:276] Timestamp: P: 1736542622630139 usec, L: 0
0110 20:57:18.773867 (+ 469us) write_op.cc:270] Start()
0110 20:57:18.773943 (+ 76us) write_op.cc:276] Timestamp: P: 1736542622684411 usec, L: 0
0110 20:57:18.774396 (+ 453us) write_op.cc:270] Start()
0110 20:57:18.774478 (+ 82us) write_op.cc:276] Timestamp: P: 1736542622768200 usec, L: 0
0110 20:57:18.774942 (+ 464us) write_op.cc:270] Start()
0110 20:57:18.775035 (+ 93us) write_op.cc:276] Timestamp: P: 1736542622774313 usec, L: 0
0110 20:57:18.775513 (+ 478us) write_op.cc:270] Start()
0110 20:57:18.775586 (+ 73us) write_op.cc:276] Timestamp: P: 1736542622799472 usec, L: 0
0110 20:57:18.776007 (+ 421us) write_op.cc:270] Start()
0110 20:57:18.776094 (+ 87us) write_op.cc:276] Timestamp: P: 1736542622825907 usec, L: 0
0110 20:57:18.776517 (+ 423us) write_op.cc:270] Start()
0110 20:57:18.776592 (+ 75us) write_op.cc:276] Timestamp: P: 1736542622830756 usec, L: 0
0110 20:57:18.794802 (+ 18210us) write_op.cc:270] Start()
0110 20:57:18.794901 (+ 99us) write_op.cc:276] Timestamp: P: 1736542622835337 usec, L: 0
0110 20:57:18.795445 (+ 544us) write_op.cc:270] Start()
0110 20:57:18.795526 (+ 81us) write_op.cc:276] Timestamp: P: 1736542622898865 usec, L: 0
0110 20:57:18.795960 (+ 434us) write_op.cc:270] Start()
0110 20:57:18.796031 (+ 71us) write_op.cc:276] Timestamp: P: 1736542622952973 usec, L: 0
0110 20:57:18.796443 (+ 412us) write_op.cc:270] Start()
0110 20:57:18.796519 (+ 76us) write_op.cc:276] Timestamp: P: 1736542622967706 usec, L: 0
0110 20:57:18.796913 (+ 394us) write_op.cc:270] Start()
0110 20:57:18.796984 (+ 71us) write_op.cc:276] Timestamp: P: 1736542622993945 usec, L: 0
0110 20:57:18.797394 (+ 410us) write_op.cc:270] Start()
0110 20:57:18.809991 (+ 12597us) write_op.cc:276] Timestamp: P: 1736542623006519 usec, L: 0
0110 20:57:18.810518 (+ 527us) write_op.cc:270] Start()
0110 20:57:18.810593 (+ 75us) write_op.cc:276] Timestamp: P: 1736542623063162 usec, L: 0
0110 20:57:18.811087 (+ 494us) write_op.cc:270] Start()
0110 20:57:18.811264 (+ 177us) write_op.cc:276] Timestamp: P: 1736542623074741 usec, L: 0
0110 20:57:18.811693 (+ 429us) write_op.cc:270] Start()
0110 20:57:18.811792 (+ 99us) write_op.cc:276] Timestamp: P: 1736542623154960 usec, L: 0
0110 20:57:18.812240 (+ 448us) write_op.cc:270] Start()
0110 20:57:18.812319 (+ 79us) write_op.cc:276] Timestamp: P: 1736542623157127 usec, L: 0
0110 20:57:18.812722 (+ 403us) write_op.cc:270] Start()
0110 20:57:18.812792 (+ 70us) write_op.cc:276] Timestamp: P: 1736542623176227 usec, L: 0
0110 20:57:18.817093 (+ 4301us) write_op.cc:270] Start()
0110 20:57:18.817171 (+ 78us) write_op.cc:276] Timestamp: P: 1736542623264527 usec, L: 0
0110 20:57:18.817629 (+ 458us) write_op.cc:270] Start()
0110 20:57:18.817713 (+ 84us) write_op.cc:276] Timestamp: P: 1736542623269003 usec, L: 0
0110 20:57:18.818128 (+ 415us) write_op.cc:270] Start()
0110 20:57:18.818198 (+ 70us) write_op.cc:276] Timestamp: P: 1736542623346740 usec, L: 0
0110 20:57:18.818619 (+ 421us) write_op.cc:270] Start()
0110 20:57:18.818690 (+ 71us) write_op.cc:276] Timestamp: P: 1736542623385212 usec, L: 0
0110 20:57:18.819094 (+ 404us) write_op.cc:270] Start()
0110 20:57:18.819161 (+ 67us) write_op.cc:276] Timestamp: P: 1736542623420443 usec, L: 0
0110 20:57:18.819571 (+ 410us) write_op.cc:270] Start()
0110 20:57:18.819653 (+ 82us) write_op.cc:276] Timestamp: P: 1736542623473360 usec, L: 0
0110 20:57:18.820152 (+ 499us) write_op.cc:270] Start()
0110 20:57:18.820230 (+ 78us) write_op.cc:276] Timestamp: P: 1736542623513596 usec, L: 0
0110 20:57:18.820623 (+ 393us) write_op.cc:270] Start()
0110 20:57:18.820692 (+ 69us) write_op.cc:276] Timestamp: P: 1736542623517136 usec, L: 0
0110 20:57:18.821092 (+ 400us) write_op.cc:270] Start()
0110 20:57:18.821164 (+ 72us) write_op.cc:276] Timestamp: P: 1736542623559752 usec, L: 0
0110 20:57:18.841646 (+ 20482us) write_op.cc:270] Start()
0110 20:57:18.841740 (+ 94us) write_op.cc:276] Timestamp: P: 1736542623583660 usec, L: 0
0110 20:57:18.843825 (+ 2085us) write_op.cc:270] Start()
0110 20:57:18.843908 (+ 83us) write_op.cc:276] Timestamp: P: 1736542623592197 usec, L: 0
0110 20:57:18.844388 (+ 480us) write_op.cc:270] Start()
0110 20:57:18.844459 (+ 71us) write_op.cc:276] Timestamp: P: 1736542623604363 usec, L: 0
0110 20:57:18.844874 (+ 415us) write_op.cc:270] Start()
0110 20:57:18.844946 (+ 72us) write_op.cc:276] Timestamp: P: 1736542623656892 usec, L: 0
0110 20:57:18.850691 (+ 5745us) write_op.cc:270] Start()
0110 20:57:18.850781 (+ 90us) write_op.cc:276] Timestamp: P: 1736542623686239 usec, L: 0
0110 20:57:18.851266 (+ 485us) write_op.cc:270] Start()
0110 20:57:18.851340 (+ 74us) write_op.cc:276] Timestamp: P: 1736542623708541 usec, L: 0
0110 20:57:18.851754 (+ 414us) write_op.cc:270] Start()
0110 20:57:18.851824 (+ 70us) write_op.cc:276] Timestamp: P: 1736542623760376 usec, L: 0
0110 20:57:18.852230 (+ 406us) write_op.cc:270] Start()
0110 20:57:18.852299 (+ 69us) write_op.cc:276] Timestamp: P: 1736542623768683 usec, L: 0
0110 20:57:18.852711 (+ 412us) write_op.cc:270] Start()
0110 20:57:18.852782 (+ 71us) write_op.cc:276] Timestamp: P: 1736542623823698 usec, L: 0
0110 20:57:18.853186 (+ 404us) write_op.cc:270] Start()
0110 20:57:18.853257 (+ 71us) write_op.cc:276] Timestamp: P: 1736542623855348 usec, L: 0
0110 20:57:18.861428 (+ 8171us) write_op.cc:270] Start()
0110 20:57:18.861511 (+ 83us) write_op.cc:276] Timestamp: P: 1736542623863575 usec, L: 0
0110 20:57:18.861942 (+ 431us) write_op.cc:270] Start()
0110 20:57:18.862015 (+ 73us) write_op.cc:276] Timestamp: P: 1736542623865904 usec, L: 0
0110 20:57:18.862441 (+ 426us) write_op.cc:270] Start()
0110 20:57:18.862568 (+ 127us) write_op.cc:276] Timestamp: P: 1736542623883851 usec, L: 0
0110 20:57:18.880237 (+ 17669us) write_op.cc:270] Start()
0110 20:57:18.880323 (+ 86us) write_op.cc:276] Timestamp: P: 1736542623914068 usec, L: 0
0110 20:57:18.880810 (+ 487us) write_op.cc:270] Start()
0110 20:57:18.880886 (+ 76us) write_op.cc:276] Timestamp: P: 1736542623940493 usec, L: 0
0110 20:57:18.881364 (+ 478us) write_op.cc:270] Start()
0110 20:57:18.881461 (+ 97us) write_op.cc:276] Timestamp: P: 1736542623942407 usec, L: 0
0110 20:57:18.881912 (+ 451us) write_op.cc:270] Start()
0110 20:57:18.881990 (+ 78us) write_op.cc:276] Timestamp: P: 1736542624017277 usec, L: 0
0110 20:57:18.908435 (+ 26445us) write_op.cc:270] Start()
0110 20:57:18.908530 (+ 95us) write_op.cc:276] Timestamp: P: 1736542624049949 usec, L: 0
0110 20:57:18.909068 (+ 538us) write_op.cc:270] Start()
0110 20:57:18.909146 (+ 78us) write_op.cc:276] Timestamp: P: 1736542624110202 usec, L: 0
0110 20:57:18.909616 (+ 470us) write_op.cc:270] Start()
0110 20:57:18.909694 (+ 78us) write_op.cc:276] Timestamp: P: 1736542624160147 usec, L: 0
0110 20:57:18.910166 (+ 472us) write_op.cc:270] Start()
0110 20:57:18.910246 (+ 80us) write_op.cc:276] Timestamp: P: 1736542624193986 usec, L: 0
0110 20:57:18.910691 (+ 445us) write_op.cc:270] Start()
0110 20:57:18.910767 (+ 76us) write_op.cc:276] Timestamp: P: 1736542624205133 usec, L: 0
0110 20:57:18.911243 (+ 476us) write_op.cc:270] Start()
0110 20:57:18.911318 (+ 75us) write_op.cc:276] Timestamp: P: 1736542624221050 usec, L: 0
0110 20:57:18.911837 (+ 519us) write_op.cc:270] Start()
0110 20:57:18.911913 (+ 76us) write_op.cc:276] Timestamp: P: 1736542624287202 usec, L: 0
0110 20:57:18.912343 (+ 430us) write_op.cc:270] Start()
0110 20:57:18.912415 (+ 72us) write_op.cc:276] Timestamp: P: 1736542624294069 usec, L: 0
0110 20:57:18.912834 (+ 419us) write_op.cc:270] Start()
0110 20:57:18.912921 (+ 87us) write_op.cc:276] Timestamp: P: 1736542624302267 usec, L: 0
0110 20:57:18.913464 (+ 543us) write_op.cc:270] Start()
0110 20:57:18.913539 (+ 75us) write_op.cc:276] Timestamp: P: 1736542624358594 usec, L: 0
0110 20:57:18.914021 (+ 482us) write_op.cc:270] Start()
0110 20:57:18.914106 (+ 85us) write_op.cc:276] Timestamp: P: 1736542624362284 usec, L: 0
0110 20:57:18.914541 (+ 435us) write_op.cc:270] Start()
0110 20:57:18.914620 (+ 79us) write_op.cc:276] Timestamp: P: 1736542624378506 usec, L: 0
0110 20:57:18.915186 (+ 566us) write_op.cc:270] Start()
0110 20:57:18.915268 (+ 82us) write_op.cc:276] Timestamp: P: 1736542624408139 usec, L: 0
0110 20:57:18.915704 (+ 436us) write_op.cc:270] Start()
0110 20:57:18.962435 (+ 46731us) write_op.cc:276] Timestamp: P: 1736542624418745 usec, L: 0
0110 20:57:18.962996 (+ 561us) write_op.cc:270] Start()
0110 20:57:18.963092 (+ 96us) write_op.cc:276] Timestamp: P: 1736542624456724 usec, L: 0
0110 20:57:18.963545 (+ 453us) write_op.cc:270] Start()
0110 20:57:18.963619 (+ 74us) write_op.cc:276] Timestamp: P: 1736542624531676 usec, L: 0
0110 20:57:18.964047 (+ 428us) write_op.cc:270] Start()
0110 20:57:18.964133 (+ 86us) write_op.cc:276] Timestamp: P: 1736542624546329 usec, L: 0
0110 20:57:18.964569 (+ 436us) write_op.cc:270] Start()
0110 20:57:18.964642 (+ 73us) write_op.cc:276] Timestamp: P: 1736542624651324 usec, L: 0
0110 20:57:18.965043 (+ 401us) write_op.cc:270] Start()
0110 20:57:18.965132 (+ 89us) write_op.cc:276] Timestamp: P: 1736542624656469 usec, L: 0
0110 20:57:18.965594 (+ 462us) write_op.cc:270] Start()
0110 20:57:18.965680 (+ 86us) write_op.cc:276] Timestamp: P: 1736542624658203 usec, L: 0
0110 20:57:18.966116 (+ 436us) write_op.cc:270] Start()
0110 20:57:18.966189 (+ 73us) write_op.cc:276] Timestamp: P: 1736542624697553 usec, L: 0
0110 20:57:18.966606 (+ 417us) write_op.cc:270] Start()
0110 20:57:18.966672 (+ 66us) write_op.cc:276] Timestamp: P: 1736542624702138 usec, L: 0
0110 20:57:18.967102 (+ 430us) write_op.cc:270] Start()
0110 20:57:18.967172 (+ 70us) write_op.cc:276] Timestamp: P: 1736542624911673 usec, L: 0
0110 20:57:18.967594 (+ 422us) write_op.cc:270] Start()
0110 20:57:18.967669 (+ 75us) write_op.cc:276] Timestamp: P: 1736542624969877 usec, L: 0
0110 20:57:18.968112 (+ 443us) write_op.cc:270] Start()
0110 20:57:18.968186 (+ 74us) write_op.cc:276] Timestamp: P: 1736542625042572 usec, L: 0
0110 20:57:18.968621 (+ 435us) write_op.cc:270] Start()
0110 20:57:18.968700 (+ 79us) write_op.cc:276] Timestamp: P: 1736542625059191 usec, L: 0
0110 20:57:18.969168 (+ 468us) write_op.cc:270] Start()
0110 20:57:18.999710 (+ 30542us) write_op.cc:276] Timestamp: P: 1736542625115245 usec, L: 0
0110 20:57:19.000285 (+ 575us) write_op.cc:270] Start()
0110 20:57:19.000372 (+ 87us) write_op.cc:276] Timestamp: P: 1736542625123307 usec, L: 0
0110 20:57:19.000830 (+ 458us) write_op.cc:270] Start()
0110 20:57:19.000912 (+ 82us) write_op.cc:276] Timestamp: P: 1736542625133957 usec, L: 0
0110 20:57:19.001360 (+ 448us) write_op.cc:270] Start()
0110 20:57:19.001455 (+ 95us) write_op.cc:276] Timestamp: P: 1736542625145055 usec, L: 0
0110 20:57:19.001902 (+ 447us) write_op.cc:270] Start()
0110 20:57:19.001977 (+ 75us) write_op.cc:276] Timestamp: P: 1736542625150092 usec, L: 0
0110 20:57:19.002432 (+ 455us) write_op.cc:270] Start()
0110 20:57:19.002508 (+ 76us) write_op.cc:276] Timestamp: P: 1736542625294665 usec, L: 0
0110 20:57:19.002950 (+ 442us) write_op.cc:270] Start()
0110 20:57:19.003025 (+ 75us) write_op.cc:276] Timestamp: P: 1736542625296442 usec, L: 0
0110 20:57:19.003475 (+ 450us) write_op.cc:270] Start()
0110 20:57:19.003557 (+ 82us) write_op.cc:276] Timestamp: P: 1736542625303304 usec, L: 0
0110 20:57:19.004007 (+ 450us) write_op.cc:270] Start()
0110 20:57:19.004092 (+ 85us) write_op.cc:276] Timestamp: P: 1736542625309984 usec, L: 0
0110 20:57:19.004526 (+ 434us) write_op.cc:270] Start()
0110 20:57:19.004604 (+ 78us) write_op.cc:276] Timestamp: P: 1736542625409205 usec, L: 0
0110 20:57:19.211872 (+207268us) write_op.cc:270] Start()
0110 20:57:19.211969 (+ 97us) write_op.cc:276] Timestamp: P: 1736542625435196 usec, L: 0
0110 20:57:19.212466 (+ 497us) write_op.cc:270] Start()
0110 20:57:19.212543 (+ 77us) write_op.cc:276] Timestamp: P: 1736542625497022 usec, L: 0
0110 20:57:19.212978 (+ 435us) write_op.cc:270] Start()
0110 20:57:19.213047 (+ 69us) write_op.cc:276] Timestamp: P: 1736542625591002 usec, L: 0
0110 20:57:19.213510 (+ 463us) write_op.cc:270] Start()
0110 20:57:19.213580 (+ 70us) write_op.cc:276] Timestamp: P: 1736542625592980 usec, L: 0
0110 20:57:19.213988 (+ 408us) write_op.cc:270] Start()
0110 20:57:19.214057 (+ 69us) write_op.cc:276] Timestamp: P: 1736542625648227 usec, L: 0
0110 20:57:19.214472 (+ 415us) write_op.cc:270] Start()
0110 20:57:19.214643 (+ 171us) write_op.cc:276] Timestamp: P: 1736542625650772 usec, L: 0
0110 20:57:19.215057 (+ 414us) write_op.cc:270] Start()
0110 20:57:19.215132 (+ 75us) write_op.cc:276] Timestamp: P: 1736542625652644 usec, L: 0
0110 20:57:19.215555 (+ 423us) write_op.cc:270] Start()
0110 20:57:19.215625 (+ 70us) write_op.cc:276] Timestamp: P: 1736542625657482 usec, L: 0
0110 20:57:19.225611 (+ 9986us) write_op.cc:270] Start()
0110 20:57:19.225691 (+ 80us) write_op.cc:276] Timestamp: P: 1736542625662074 usec, L: 0
0110 20:57:19.226163 (+ 472us) write_op.cc:270] Start()
0110 20:57:19.226362 (+ 199us) write_op.cc:276] Timestamp: P: 1736542625693590 usec, L: 0
0110 20:57:19.226928 (+ 566us) write_op.cc:270] Start()
0110 20:57:19.227117 (+ 189us) write_op.cc:276] Timestamp: P: 1736542625750283 usec, L: 0
0110 20:57:19.227598 (+ 481us) write_op.cc:270] Start()
0110 20:57:19.227675 (+ 77us) write_op.cc:276] Timestamp: P: 1736542625831293 usec, L: 0
0110 20:57:19.328534 (+100859us) write_op.cc:270] Start()
0110 20:57:19.328630 (+ 96us) write_op.cc:276] Timestamp: P: 1736542625915446 usec, L: 0
0110 20:57:19.329134 (+ 504us) write_op.cc:270] Start()
0110 20:57:19.329227 (+ 93us) write_op.cc:276] Timestamp: P: 1736542625922854 usec, L: 0
0110 20:57:19.329756 (+ 529us) write_op.cc:270] Start()
0110 20:57:19.329842 (+ 86us) write_op.cc:276] Timestamp: P: 1736542625933304 usec, L: 0
0110 20:57:19.330278 (+ 436us) write_op.cc:270] Start()
0110 20:57:19.330368 (+ 90us) write_op.cc:276] Timestamp: P: 1736542625944279 usec, L: 0
0110 20:57:19.330806 (+ 438us) write_op.cc:270] Start()
0110 20:57:19.330877 (+ 71us) write_op.cc:276] Timestamp: P: 1736542625947343 usec, L: 0
0110 20:57:19.348564 (+ 17687us) write_op.cc:270] Start()
0110 20:57:19.348650 (+ 86us) write_op.cc:276] Timestamp: P: 1736542625959442 usec, L: 0
0110 20:57:19.349140 (+ 490us) write_op.cc:270] Start()
0110 20:57:19.349207 (+ 67us) write_op.cc:276] Timestamp: P: 1736542626000897 usec, L: 0
0110 20:57:19.349680 (+ 473us) write_op.cc:270] Start()
0110 20:57:19.349751 (+ 71us) write_op.cc:276] Timestamp: P: 1736542626013251 usec, L: 0
0110 20:57:19.350160 (+ 409us) write_op.cc:270] Start()
0110 20:57:19.350226 (+ 66us) write_op.cc:276] Timestamp: P: 1736542626024892 usec, L: 0
0110 20:57:19.379903 (+ 29677us) write_op.cc:270] Start()
0110 20:57:19.379983 (+ 80us) write_op.cc:276] Timestamp: P: 1736542626078758 usec, L: 0
0110 20:57:19.393657 (+ 13674us) write_op.cc:270] Start()
0110 20:57:19.393733 (+ 76us) write_op.cc:276] Timestamp: P: 1736542626100560 usec, L: 0
0110 20:57:19.394205 (+ 472us) write_op.cc:270] Start()
0110 20:57:19.394276 (+ 71us) write_op.cc:276] Timestamp: P: 1736542626207296 usec, L: 0
0110 20:57:19.394676 (+ 400us) write_op.cc:270] Start()
0110 20:57:19.394739 (+ 63us) write_op.cc:276] Timestamp: P: 1736542626223376 usec, L: 0
0110 20:57:19.395152 (+ 413us) write_op.cc:270] Start()
0110 20:57:19.395217 (+ 65us) write_op.cc:276] Timestamp: P: 1736542626249286 usec, L: 0
0110 20:57:19.395630 (+ 413us) write_op.cc:270] Start()
0110 20:57:19.395700 (+ 70us) write_op.cc:276] Timestamp: P: 1736542626253109 usec, L: 0
0110 20:57:19.396114 (+ 414us) write_op.cc:270] Start()
0110 20:57:19.396179 (+ 65us) write_op.cc:276] Timestamp: P: 1736542626266973 usec, L: 0
0110 20:57:19.396587 (+ 408us) write_op.cc:270] Start()
0110 20:57:19.396651 (+ 64us) write_op.cc:276] Timestamp: P: 1736542626275717 usec, L: 0
0110 20:57:19.397048 (+ 397us) write_op.cc:270] Start()
0110 20:57:19.397113 (+ 65us) write_op.cc:276] Timestamp: P: 1736542626283395 usec, L: 0
0110 20:57:19.465140 (+ 68027us) write_op.cc:270] Start()
0110 20:57:19.465236 (+ 96us) write_op.cc:276] Timestamp: P: 1736542626378610 usec, L: 0
0110 20:57:19.465787 (+ 551us) write_op.cc:270] Start()
0110 20:57:19.465855 (+ 68us) write_op.cc:276] Timestamp: P: 1736542626401309 usec, L: 0
0110 20:57:19.466235 (+ 380us) write_op.cc:270] Start()
0110 20:57:19.466304 (+ 69us) write_op.cc:276] Timestamp: P: 1736542626440162 usec, L: 0
0110 20:57:19.466705 (+ 401us) write_op.cc:270] Start()
0110 20:57:19.466769 (+ 64us) write_op.cc:276] Timestamp: P: 1736542626508965 usec, L: 0
0110 20:57:19.474038 (+ 7269us) write_op.cc:270] Start()
0110 20:57:19.474121 (+ 83us) write_op.cc:276] Timestamp: P: 1736542626523186 usec, L: 0
0110 20:57:19.474559 (+ 438us) write_op.cc:270] Start()
0110 20:57:19.474627 (+ 68us) write_op.cc:276] Timestamp: P: 1736542626524848 usec, L: 0
0110 20:57:19.475016 (+ 389us) write_op.cc:270] Start()
0110 20:57:19.475102 (+ 86us) write_op.cc:276] Timestamp: P: 1736542626562045 usec, L: 0
0110 20:57:19.475534 (+ 432us) write_op.cc:270] Start()
0110 20:57:19.475601 (+ 67us) write_op.cc:276] Timestamp: P: 1736542626621936 usec, L: 0
0110 20:57:19.476001 (+ 400us) write_op.cc:270] Start()
0110 20:57:19.476066 (+ 65us) write_op.cc:276] Timestamp: P: 1736542626633100 usec, L: 0
0110 20:57:19.476481 (+ 415us) write_op.cc:270] Start()
0110 20:57:19.476546 (+ 65us) write_op.cc:276] Timestamp: P: 1736542626680356 usec, L: 0
0110 20:57:19.477081 (+ 535us) write_op.cc:270] Start()
0110 20:57:19.477159 (+ 78us) write_op.cc:276] Timestamp: P: 1736542626698103 usec, L: 0
0110 20:57:19.477668 (+ 509us) write_op.cc:270] Start()
0110 20:57:19.477738 (+ 70us) write_op.cc:276] Timestamp: P: 1736542626703913 usec, L: 0
0110 20:57:19.478144 (+ 406us) write_op.cc:270] Start()
0110 20:57:19.478209 (+ 65us) write_op.cc:276] Timestamp: P: 1736542626827667 usec, L: 0
0110 20:57:19.478591 (+ 382us) write_op.cc:270] Start()
0110 20:57:19.478654 (+ 63us) write_op.cc:276] Timestamp: P: 1736542626830444 usec, L: 0
0110 20:57:19.479025 (+ 371us) write_op.cc:270] Start()
0110 20:57:19.479099 (+ 74us) write_op.cc:276] Timestamp: P: 1736542626832231 usec, L: 0
0110 20:57:19.479500 (+ 401us) write_op.cc:270] Start()
0110 20:57:19.479569 (+ 69us) write_op.cc:276] Timestamp: P: 1736542626833964 usec, L: 0
0110 20:57:19.483221 (+ 3652us) write_op.cc:270] Start()
0110 20:57:19.483304 (+ 83us) write_op.cc:276] Timestamp: P: 1736542627064050 usec, L: 0
0110 20:57:19.483717 (+ 413us) write_op.cc:270] Start()
0110 20:57:19.483777 (+ 60us) write_op.cc:276] Timestamp: P: 1736542627128423 usec, L: 0
0110 20:57:19.485579 (+ 1802us) write_op.cc:270] Start()
0110 20:57:19.485656 (+ 77us) write_op.cc:276] Timestamp: P: 1736542627200648 usec, L: 0
0110 20:57:19.486086 (+ 430us) write_op.cc:270] Start()
0110 20:57:19.486154 (+ 68us) write_op.cc:276] Timestamp: P: 1736542627347109 usec, L: 0
0110 20:57:19.488165 (+ 2011us) write_op.cc:270] Start()
0110 20:57:19.488246 (+ 81us) write_op.cc:276] Timestamp: P: 1736542627349849 usec, L: 0
0110 20:57:19.488674 (+ 428us) write_op.cc:270] Start()
0110 20:57:19.488741 (+ 67us) write_op.cc:276] Timestamp: P: 1736542627390188 usec, L: 0
0110 20:57:19.489155 (+ 414us) write_op.cc:270] Start()
0110 20:57:19.489224 (+ 69us) write_op.cc:276] Timestamp: P: 1736542627399527 usec, L: 0
0110 20:57:19.489647 (+ 423us) write_op.cc:270] Start()
0110 20:57:19.489711 (+ 64us) write_op.cc:276] Timestamp: P: 1736542627410567 usec, L: 0
0110 20:57:19.490661 (+ 950us) write_op.cc:270] Start()
0110 20:57:19.490725 (+ 64us) write_op.cc:276] Timestamp: P: 1736542627418557 usec, L: 0
0110 20:57:19.516734 (+ 26009us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b5400010280. stack: 00007f62bccea853 00007f62bccea750 00007f62bc85f584 00000000004fcd24 00000000004fccb9 00007f62c4001a81 00007f62c40d59bc 00007f62c40cb5dd 00007f62c408ef27 00007f62c408e9ff 00007f62c3c5c040 00007f62c3c65536 00007f62c3c6499e 00007f62c4c4b2d9 00007f62c3417694 00007f62c3417622
0110 20:57:19.517300 (+ 566us) write_op.cc:270] Start()
0110 20:57:19.517380 (+ 80us) write_op.cc:276] Timestamp: P: 1736542627446606 usec, L: 0
0110 20:57:19.517830 (+ 450us) write_op.cc:270] Start()
0110 20:57:19.517895 (+ 65us) write_op.cc:276] Timestamp: P: 1736542627503048 usec, L: 0
0110 20:57:19.518308 (+ 413us) write_op.cc:270] Start()
0110 20:57:19.518374 (+ 66us) write_op.cc:276] Timestamp: P: 1736542627505982 usec, L: 0
0110 20:57:19.518756 (+ 382us) write_op.cc:270] Start()
0110 20:57:19.518822 (+ 66us) write_op.cc:276] Timestamp: P: 1736542627547850 usec, L: 0
0110 20:57:19.519253 (+ 431us) write_op.cc:270] Start()
0110 20:57:19.519316 (+ 63us) write_op.cc:276] Timestamp: P: 1736542627697545 usec, L: 0
0110 20:57:19.519683 (+ 367us) write_op.cc:270] Start()
0110 20:57:19.519744 (+ 61us) write_op.cc:276] Timestamp: P: 1736542627702489 usec, L: 0
0110 20:57:19.520128 (+ 384us) write_op.cc:270] Start()
0110 20:57:19.520198 (+ 70us) write_op.cc:276] Timestamp: P: 1736542627825424 usec, L: 0
0110 20:57:19.520551 (+ 353us) write_op.cc:270] Start()
0110 20:57:19.520611 (+ 60us) write_op.cc:276] Timestamp: P: 1736542627932796 usec, L: 0
0110 20:57:19.520957 (+ 346us) write_op.cc:270] Start()
0110 20:57:19.521019 (+ 62us) write_op.cc:276] Timestamp: P: 1736542627981470 usec, L: 0
0110 20:57:19.521428 (+ 409us) write_op.cc:270] Start()
0110 20:57:19.521491 (+ 63us) write_op.cc:276] Timestamp: P: 1736542627992904 usec, L: 0
0110 20:57:19.521860 (+ 369us) write_op.cc:270] Start()
0110 20:57:19.521921 (+ 61us) write_op.cc:276] Timestamp: P: 1736542628234506 usec, L: 0
0110 20:57:19.522711 (+ 790us) write_op.cc:270] Start()
0110 20:57:19.522776 (+ 65us) write_op.cc:276] Timestamp: P: 1736542628236211 usec, L: 0
0110 20:57:19.523180 (+ 404us) write_op.cc:270] Start()
0110 20:57:19.523243 (+ 63us) write_op.cc:276] Timestamp: P: 1736542628239461 usec, L: 0
0110 20:57:19.523624 (+ 381us) write_op.cc:270] Start()
0110 20:57:19.523692 (+ 68us) write_op.cc:276] Timestamp: P: 1736542628241039 usec, L: 0
0110 20:57:19.524096 (+ 404us) write_op.cc:270] Start()
0110 20:57:19.524161 (+ 65us) write_op.cc:276] Timestamp: P: 1736542628246439 usec, L: 0
0110 20:57:19.524547 (+ 386us) write_op.cc:270] Start()
0110 20:57:19.524610 (+ 63us) write_op.cc:276] Timestamp: P: 1736542628292578 usec, L: 0
0110 20:57:19.524994 (+ 384us) write_op.cc:270] Start()
0110 20:57:19.525062 (+ 68us) write_op.cc:276] Timestamp: P: 1736542628783112 usec, L: 0
0110 20:57:19.525614 (+ 552us) write_op.cc:270] Start()
0110 20:57:19.525679 (+ 65us) write_op.cc:276] Timestamp: P: 1736542628799979 usec, L: 0
0110 20:57:19.526345 (+ 666us) write_op.cc:270] Start()
0110 20:57:19.526481 (+ 136us) write_op.cc:276] Timestamp: P: 1736542628820676 usec, L: 0
0110 20:57:19.526873 (+ 392us) write_op.cc:270] Start()
0110 20:57:19.526937 (+ 64us) write_op.cc:276] Timestamp: P: 1736542628851864 usec, L: 0
0110 20:57:19.527347 (+ 410us) write_op.cc:270] Start()
0110 20:57:19.527422 (+ 75us) write_op.cc:276] Timestamp: P: 1736542628886285 usec, L: 0
0110 20:57:19.527809 (+ 387us) write_op.cc:270] Start()
0110 20:57:19.527869 (+ 60us) write_op.cc:276] Timestamp: P: 1736542628894049 usec, L: 0
0110 20:57:19.528278 (+ 409us) write_op.cc:270] Start()
0110 20:57:19.528347 (+ 69us) write_op.cc:276] Timestamp: P: 1736542628934530 usec, L: 0
0110 20:57:19.528746 (+ 399us) write_op.cc:270] Start()
0110 20:57:19.528807 (+ 61us) write_op.cc:276] Timestamp: P: 1736542628964343 usec, L: 0
0110 20:57:19.529200 (+ 393us) write_op.cc:270] Start()
0110 20:57:19.529267 (+ 67us) write_op.cc:276] Timestamp: P: 1736542629008637 usec, L: 0
0110 20:57:19.529676 (+ 409us) write_op.cc:270] Start()
0110 20:57:19.529740 (+ 64us) write_op.cc:276] Timestamp: P: 1736542629054215 usec, L: 0
0110 20:57:19.530149 (+ 409us) write_op.cc:270] Start()
0110 20:57:19.530212 (+ 63us) write_op.cc:276] Timestamp: P: 1736542629096636 usec, L: 0
0110 20:57:19.530603 (+ 391us) write_op.cc:270] Start()
0110 20:57:19.530664 (+ 61us) write_op.cc:276] Timestamp: P: 1736542629214455 usec, L: 0
0110 20:57:19.531086 (+ 422us) write_op.cc:270] Start()
0110 20:57:19.531148 (+ 62us) write_op.cc:276] Timestamp: P: 1736542629375001 usec, L: 0
0110 20:57:19.531518 (+ 370us) write_op.cc:270] Start()
0110 20:57:19.531579 (+ 61us) write_op.cc:276] Timestamp: P: 1736542629427464 usec, L: 0
0110 20:57:19.532116 (+ 537us) write_op.cc:270] Start()
0110 20:57:19.532183 (+ 67us) write_op.cc:276] Timestamp: P: 1736542629450112 usec, L: 0
0110 20:57:19.532607 (+ 424us) write_op.cc:270] Start()
0110 20:57:19.532676 (+ 69us) write_op.cc:276] Timestamp: P: 1736542629859281 usec, L: 0
0110 20:57:19.533067 (+ 391us) write_op.cc:270] Start()
0110 20:57:19.533146 (+ 79us) write_op.cc:276] Timestamp: P: 1736542629883977 usec, L: 0
0110 20:57:19.533859 (+ 713us) write_op.cc:270] Start()
0110 20:57:19.533924 (+ 65us) write_op.cc:276] Timestamp: P: 1736542629890753 usec, L: 0
0110 20:57:19.534333 (+ 409us) write_op.cc:270] Start()
0110 20:57:19.534396 (+ 63us) write_op.cc:276] Timestamp: P: 1736542630013283 usec, L: 0
0110 20:57:19.534774 (+ 378us) write_op.cc:270] Start()
0110 20:57:19.534835 (+ 61us) write_op.cc:276] Timestamp: P: 1736542630015141 usec, L: 0
0110 20:57:19.535235 (+ 400us) write_op.cc:270] Start()
0110 20:57:19.535306 (+ 71us) write_op.cc:276] Timestamp: P: 1736542630016773 usec, L: 0
0110 20:57:19.535676 (+ 370us) write_op.cc:270] Start()
0110 20:57:19.535738 (+ 62us) write_op.cc:276] Timestamp: P: 1736542630177539 usec, L: 0
0110 20:57:19.536136 (+ 398us) write_op.cc:270] Start()
0110 20:57:19.536198 (+ 62us) write_op.cc:276] Timestamp: P: 1736542630181926 usec, L: 0
0110 20:57:19.536562 (+ 364us) write_op.cc:270] Start()
0110 20:57:19.536629 (+ 67us) write_op.cc:276] Timestamp: P: 1736542630272715 usec, L: 0
0110 20:57:19.537005 (+ 376us) write_op.cc:270] Start()
0110 20:57:19.537065 (+ 60us) write_op.cc:276] Timestamp: P: 1736542630343020 usec, L: 0
0110 20:57:19.537553 (+ 488us) write_op.cc:270] Start()
0110 20:57:19.537741 (+ 188us) write_op.cc:276] Timestamp: P: 1736542630464806 usec, L: 0
0110 20:57:19.538161 (+ 420us) write_op.cc:270] Start()
0110 20:57:19.538224 (+ 63us) write_op.cc:276] Timestamp: P: 1736542630512925 usec, L: 0
0110 20:57:19.538602 (+ 378us) write_op.cc:270] Start()
0110 20:57:19.538666 (+ 64us) write_op.cc:276] Timestamp: P: 1736542630574534 usec, L: 0
0110 20:57:19.539129 (+ 463us) write_op.cc:270] Start()
0110 20:57:19.539203 (+ 74us) write_op.cc:276] Timestamp: P: 1736542630577090 usec, L: 0
0110 20:57:19.539588 (+ 385us) write_op.cc:270] Start()
0110 20:57:19.539653 (+ 65us) write_op.cc:276] Timestamp: P: 1736542630648919 usec, L: 0
0110 20:57:19.540247 (+ 594us) write_op.cc:270] Start()
0110 20:57:19.540310 (+ 63us) write_op.cc:276] Timestamp: P: 1736542630762419 usec, L: 0
0110 20:57:19.540715 (+ 405us) write_op.cc:270] Start()
0110 20:57:19.540784 (+ 69us) write_op.cc:276] Timestamp: P: 1736542630897030 usec, L: 0
0110 20:57:19.541198 (+ 414us) write_op.cc:270] Start()
0110 20:57:19.541262 (+ 64us) write_op.cc:276] Timestamp: P: 1736542630900013 usec, L: 0
0110 20:57:19.541677 (+ 415us) write_op.cc:270] Start()
0110 20:57:19.541740 (+ 63us) write_op.cc:276] Timestamp: P: 1736542630996521 usec, L: 0
0110 20:57:19.542122 (+ 382us) write
I20250110 20:57:24.491154 5600 tablet_bootstrap.cc:492] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:24.496253 5600 tablet_bootstrap.cc:654] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.512250 5594 tablet_bootstrap.cc:492] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.512651 5594 ts_tablet_manager.cc:1397] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.157s user 0.008s sys 0.000s
I20250110 20:57:24.514850 5600 tablet_bootstrap.cc:492] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:24.514736 5594 raft_consensus.cc:357] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.515277 5600 ts_tablet_manager.cc:1397] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.003s
I20250110 20:57:24.515358 5594 raft_consensus.cc:383] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.515621 5594 raft_consensus.cc:738] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.517562 5600 raft_consensus.cc:357] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.518282 5600 raft_consensus.cc:383] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.518576 5600 raft_consensus.cc:738] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.519129 5600 consensus_queue.cc:260] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.521440 5600 ts_tablet_manager.cc:1428] T c7961f8bd3ff42ffab214a04c704fd26 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20250110 20:57:24.522467 4884 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb (table default.Table_91e7b8f8b37f49ee8fc122bcec6530c0 [id=4bc4846633fc4e74b4d2e79c01e3db99], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 57a9b7c9cd3c4868b77fcf5e4b6801bb consensus state has no leader
I20250110 20:57:24.525664 5594 consensus_queue.cc:260] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.527889 5600 tablet_bootstrap.cc:492] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
I20250110 20:57:24.532621 6464 raft_consensus.cc:491] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:24.533008 6464 raft_consensus.cc:513] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.534777 6464 leader_election.cc:290] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:24.536067 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758430b311a8478c9cab2bcf79253b52" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384" is_pre_election: true
I20250110 20:57:24.536625 5318 raft_consensus.cc:2463] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 0.
I20250110 20:57:24.536862 6540 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b84772eba3f0460faf8523f76eb57e53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } } }
I20250110 20:57:24.537629 6540 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:24.537986 5398 leader_election.cc:304] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:24.538712 6464 raft_consensus.cc:2798] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:24.539052 6464 raft_consensus.cc:491] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:24.539394 6464 raft_consensus.cc:3054] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:24.544013 5592 ts_tablet_manager.cc:1918] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:24.544361 5592 log.cc:1198] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/fcc7b1acf4114bf69d801437855760df
I20250110 20:57:24.551646 5594 ts_tablet_manager.cc:1428] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20250110 20:57:24.552817 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 33db8a2fdd9c4fde985345604fd04215 (table default.Table_01a45b23c3694bd991e5a47e493e3b4d [id=505b9ca1a68e401aa6f6f28a41f09181]). Expected version 2 got 0
I20250110 20:57:24.561444 5594 tablet_bootstrap.cc:492] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:24.581122 6464 raft_consensus.cc:513] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.582383 5600 tablet_bootstrap.cc:654] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.544999 5592 ts_tablet_manager.cc:1939] T fcc7b1acf4114bf69d801437855760df P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:24.584785 4897 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet 33db8a2fdd9c4fde985345604fd04215 (table default.Table_01a45b23c3694bd991e5a47e493e3b4d [id=505b9ca1a68e401aa6f6f28a41f09181]). Expected version 2 got 0
I20250110 20:57:24.586300 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 266) took 5457 ms. Trace:
I20250110 20:57:24.586591 5592 rpcz_store.cc:276] 0110 20:57:19.129360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.265265 (+135905us) service_pool.cc:225] Handling call
0110 20:57:24.366800 (+5101535us) ts_tablet_manager.cc:1145] Deleting tablet fcc7b1acf4114bf69d801437855760df
0110 20:57:24.366844 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.472901 (+106057us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.543822 (+ 70921us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.586246 (+ 42424us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5101057}
I20250110 20:57:24.587563 5592 tablet_replica.cc:331] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:24.588081 5318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "758430b311a8478c9cab2bcf79253b52" candidate_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46005a5920cc4a3697a5c61b1c271384"
I20250110 20:57:24.588621 5318 raft_consensus.cc:3054] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:24.590319 5594 tablet_bootstrap.cc:654] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.594103 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet fcc7b1acf4114bf69d801437855760df (table default.Table_3023e8331a774a0e911891f54fe766c1 [id=cd444050d75c46b98d6ab0ae29f62110]) successfully deleted
I20250110 20:57:24.595701 5318 raft_consensus.cc:2463] T 758430b311a8478c9cab2bcf79253b52 P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3c5bff1e70804ab4a1bb63bdb9345ce6 in term 1.
I20250110 20:57:24.596688 5398 leader_election.cc:304] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:24.597350 6356 raft_consensus.cc:2798] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:24.597787 6356 raft_consensus.cc:695] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Becoming Leader. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Running, Role: LEADER
I20250110 20:57:24.598537 6356 consensus_queue.cc:237] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.605525 6464 leader_election.cc:290] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
I20250110 20:57:24.607568 6205 raft_consensus.cc:491] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:24.607961 6205 raft_consensus.cc:513] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.609746 6205 leader_election.cc:290] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:24.611929 5398 leader_election.cc:343] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Not found: Tablet not found: 7df3aafea18f49249cceb8a9431064e6
I20250110 20:57:24.611917 5289 tablet_service.cc:1514] Processing DeleteTablet for tablet c7961f8bd3ff42ffab214a04c704fd26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:17 UTC) from {username='slave'} at 127.0.0.1:55028
I20250110 20:57:24.588244 5592 raft_consensus.cc:2238] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:24.621744 5592 raft_consensus.cc:2267] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:24.623510 5592 ts_tablet_manager.cc:1905] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:24.584053 6454 ts_tablet_manager.cc:1918] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:24.629606 6454 log.cc:1198] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/4b28befa048244309a06f69b8a3b4b8f
I20250110 20:57:24.630273 6454 ts_tablet_manager.cc:1939] T 4b28befa048244309a06f69b8a3b4b8f P 46005a5920cc4a3697a5c61b1c271384: Deleting consensus metadata
I20250110 20:57:24.631847 6454 tablet_replica.cc:331] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384: stopping tablet replica
I20250110 20:57:24.631968 4886 catalog_manager.cc:4872] TS 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): tablet 4b28befa048244309a06f69b8a3b4b8f (table default.Table_033612db3a3843f999da6a5beb2b8b47 [id=6876fd7b8dd8452b94c04eac6be82cbb]) successfully deleted
I20250110 20:57:24.632671 6454 raft_consensus.cc:2238] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:24.633114 6454 raft_consensus.cc:2267] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:24.637385 4900 catalog_manager.cc:5526] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:24.643272 5594 tablet_bootstrap.cc:492] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.643643 5594 ts_tablet_manager.cc:1397] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.082s user 0.010s sys 0.003s
I20250110 20:57:24.645637 5594 raft_consensus.cc:357] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.646224 5594 raft_consensus.cc:383] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.646442 5594 raft_consensus.cc:738] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.646941 5594 consensus_queue.cc:260] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.649049 5594 ts_tablet_manager.cc:1428] T a687769f69d844cc8564e61dc324d2b2 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20250110 20:57:24.649856 5594 tablet_bootstrap.cc:492] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:24.671126 5594 tablet_bootstrap.cc:654] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.685247 6454 ts_tablet_manager.cc:1905] T a95f05fc5afe4877a606b501dbd09551 P 46005a5920cc4a3697a5c61b1c271384: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:24.693116 5594 tablet_bootstrap.cc:492] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.693514 5594 ts_tablet_manager.cc:1397] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.010s
I20250110 20:57:24.694588 5600 tablet_bootstrap.cc:492] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384: No bootstrap required, opened a new log
I20250110 20:57:24.694973 5600 ts_tablet_manager.cc:1397] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384: Time spent bootstrapping tablet: real 0.173s user 0.014s sys 0.001s
I20250110 20:57:24.696177 6547 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b84772eba3f0460faf8523f76eb57e53)
I20250110 20:57:24.696900 6547 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "63aa77ee6ec84a6e82132b2de9e77486" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 34617 } } peers { permanent_uuid: "b84772eba3f0460faf8523f76eb57e53" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 34381 } } peers { permanent_uuid: "14d1e2d5c72f498f915c367e8750361f" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 37233 } }
I20250110 20:57:24.695571 5594 raft_consensus.cc:357] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.698273 5594 raft_consensus.cc:383] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.698549 5594 raft_consensus.cc:738] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.699206 5594 consensus_queue.cc:260] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.701124 5594 ts_tablet_manager.cc:1428] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20250110 20:57:24.704574 6532 raft_consensus.cc:491] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:24.704986 6532 raft_consensus.cc:513] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.706754 6532 leader_election.cc:290] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
W20250110 20:57:24.708537 5257 connection.cc:537] client connection to 127.4.171.193:36063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:24.708945 5257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 20:57:24.714097 5594 tablet_bootstrap.cc:492] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
W20250110 20:57:24.714486 4887 connection.cc:537] client connection to 127.4.171.193:36063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:24.714831 4887 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 3 similar messages]
I20250110 20:57:24.717079 6547 leader_election.cc:290] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b84772eba3f0460faf8523f76eb57e53 (127.4.171.253:34381), 14d1e2d5c72f498f915c367e8750361f (127.4.171.252:37233)
I20250110 20:57:24.718318 4910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 3 candidate_status { last_received { term: 2 index: 404 } } ignore_live_leader: false dest_uuid: "b84772eba3f0460faf8523f76eb57e53" is_pre_election: true
I20250110 20:57:24.719252 5594 tablet_bootstrap.cc:654] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.724610 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80685cb3c494283bea421195bf7751f" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" is_pre_election: true
I20250110 20:57:24.725189 5457 raft_consensus.cc:2463] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 0.
I20250110 20:57:24.697062 5600 raft_consensus.cc:357] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.725754 5600 raft_consensus.cc:383] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.726002 5600 raft_consensus.cc:738] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46005a5920cc4a3697a5c61b1c271384, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.726276 5255 leader_election.cc:304] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:24.726554 5600 consensus_queue.cc:260] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.726984 6532 raft_consensus.cc:2798] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:24.727293 6532 raft_consensus.cc:491] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:24.727627 6532 raft_consensus.cc:3054] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:24.728722 5600 ts_tablet_manager.cc:1428] T 26e45db2f7db4c74bab4367857412ec5 P 46005a5920cc4a3697a5c61b1c271384: Time spent starting tablet: real 0.034s user 0.005s sys 0.001s
I20250110 20:57:24.734766 6532 raft_consensus.cc:513] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.737699 5457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f80685cb3c494283bea421195bf7751f" candidate_uuid: "46005a5920cc4a3697a5c61b1c271384" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6"
I20250110 20:57:24.738232 5457 raft_consensus.cc:3054] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:24.744853 5457 raft_consensus.cc:2463] T f80685cb3c494283bea421195bf7751f P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46005a5920cc4a3697a5c61b1c271384 in term 1.
I20250110 20:57:24.745956 5255 leader_election.cc:304] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6, 46005a5920cc4a3697a5c61b1c271384; no voters:
I20250110 20:57:24.746649 6531 raft_consensus.cc:2798] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:24.750105 5600 tablet_bootstrap.cc:492] T ba0d83a50ec0479f9673a57370b0e879 P 46005a5920cc4a3697a5c61b1c271384: Bootstrap starting.
W20250110 20:57:24.754829 4886 connection.cc:537] server connection from 127.4.171.193:46155 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:24.755704 5399 connection.cc:537] client connection to 127.4.171.193:36063 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:24.756094 5399 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250110 20:57:24.756780 6532 leader_election.cc:290] T f80685cb3c494283bea421195bf7751f P 46005a5920cc4a3697a5c61b1c271384 [CANDIDATE]: Term 1 election: Requested vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843)
I20250110 20:57:24.766031 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 5229
W20250110 20:57:24.769222 4887 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.171.193:36063, user_credentials={real_user=slave}} blocked reactor thread for 54381.1us
I20250110 20:57:24.777806 5592 ts_tablet_manager.cc:1918] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:24.778194 5592 log.cc:1198] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/5c5b954dbccb45b2a6ab7cc5cceadffb
I20250110 20:57:24.778885 5592 ts_tablet_manager.cc:1939] T 5c5b954dbccb45b2a6ab7cc5cceadffb P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:24.780759 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 267) took 5651 ms. Trace:
I20250110 20:57:24.781036 5592 rpcz_store.cc:276] 0110 20:57:19.129706 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.265782 (+136076us) service_pool.cc:225] Handling call
0110 20:57:24.587423 (+5321641us) ts_tablet_manager.cc:1145] Deleting tablet 5c5b954dbccb45b2a6ab7cc5cceadffb
0110 20:57:24.587467 (+ 44us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.706145 (+118678us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.757265 (+ 51120us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.780708 (+ 23443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5321149}
I20250110 20:57:24.782196 5592 tablet_replica.cc:331] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
I20250110 20:57:24.782328 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 5c5b954dbccb45b2a6ab7cc5cceadffb (table default.Table_b89da7c5e86341c39853bbacce851ad4 [id=7833678313fa4c76ad1eb6d19315bb23]) successfully deleted
I20250110 20:57:24.783030 5592 raft_consensus.cc:2238] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:24.783456 5592 raft_consensus.cc:2267] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:24.785169 5592 ts_tablet_manager.cc:1905] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:24.791394 5594 tablet_bootstrap.cc:492] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.791769 5594 ts_tablet_manager.cc:1397] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.001s
I20250110 20:57:24.793864 5594 raft_consensus.cc:357] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.794466 5594 raft_consensus.cc:383] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.794694 5594 raft_consensus.cc:738] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.795259 5594 consensus_queue.cc:260] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.804298 6120 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "63aa77ee6ec84a6e82132b2de9e77486" candidate_term: 3 candidate_status { last_received { term: 2 index: 404 } } ignore_live_leader: false dest_uuid: "14d1e2d5c72f498f915c367e8750361f" is_pre_election: true
I20250110 20:57:24.806420 4812 leader_election.cc:304] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 63aa77ee6ec84a6e82132b2de9e77486; no voters: 14d1e2d5c72f498f915c367e8750361f, b84772eba3f0460faf8523f76eb57e53
W20250110 20:57:24.811049 4887 catalog_manager.cc:4615] TS ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 017a93be79ad49f0a72857cae4ada768: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111)
I20250110 20:57:24.811846 6547 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 63aa77ee6ec84a6e82132b2de9e77486 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250110 20:57:24.816588 5594 ts_tablet_manager.cc:1428] T 1bda90a78a5b4e2a839d496b9317f6c0 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20250110 20:57:24.826123 5594 tablet_bootstrap.cc:492] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
W20250110 20:57:24.842799 5399 consensus_peers.cc:487] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.843176 5399 leader_election.cc:336] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111)
W20250110 20:57:24.843844 5399 leader_election.cc:336] T 484a3aebaafb4f2aa47f009b773209fe P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111)
I20250110 20:57:24.849838 5592 ts_tablet_manager.cc:1918] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:24.850179 5592 log.cc:1198] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/637a39e4264b40dd9480dc13e37359d9
I20250110 20:57:24.850826 5592 ts_tablet_manager.cc:1939] T 637a39e4264b40dd9480dc13e37359d9 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting consensus metadata
I20250110 20:57:24.852649 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 268) took 5717 ms. Trace:
I20250110 20:57:24.852936 5592 rpcz_store.cc:276] 0110 20:57:19.134924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.266311 (+131387us) service_pool.cc:225] Handling call
0110 20:57:24.782039 (+5515728us) ts_tablet_manager.cc:1145] Deleting tablet 637a39e4264b40dd9480dc13e37359d9
0110 20:57:24.782094 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.833570 (+ 51476us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.849643 (+ 16073us) tablet_metadata.cc:641] Metadata flushed
0110 20:57:24.852599 (+ 2956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5515291}
I20250110 20:57:24.854068 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 269) took 5718 ms. Trace:
I20250110 20:57:24.854166 4885 catalog_manager.cc:4872] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): tablet 637a39e4264b40dd9480dc13e37359d9 (table default.Table_638a8034373e4e3bbfb842beaacdaab8 [id=42aa140491c94323b9a140eea45f28de]) successfully deleted
I20250110 20:57:24.854346 5592 rpcz_store.cc:276] 0110 20:57:19.135214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.266779 (+131565us) service_pool.cc:225] Handling call
0110 20:57:24.853797 (+5587018us) ts_tablet_manager.cc:1145] Deleting tablet 3a520230d9fd4a74914e46f784dcf747
0110 20:57:24.853852 (+ 55us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.854025 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5586553}
W20250110 20:57:24.855428 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 3a520230d9fd4a74914e46f784dcf747 because the tablet was not found. No further retry: Not found: Tablet not found: 3a520230d9fd4a74914e46f784dcf747
I20250110 20:57:24.855842 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 270) took 5720 ms. Trace:
I20250110 20:57:24.856989 5592 rpcz_store.cc:276] 0110 20:57:19.135520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.267279 (+131759us) service_pool.cc:225] Handling call
0110 20:57:24.855076 (+5587797us) ts_tablet_manager.cc:1145] Deleting tablet da24904dece04a6194790054f6d7153a
0110 20:57:24.855126 (+ 50us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.855799 (+ 673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5587361}
W20250110 20:57:24.858042 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet da24904dece04a6194790054f6d7153a because the tablet was not found. No further retry: Not found: Tablet not found: da24904dece04a6194790054f6d7153a
I20250110 20:57:24.857995 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 263) took 5729 ms. Trace:
I20250110 20:57:24.859424 5592 rpcz_store.cc:276] 0110 20:57:19.128543 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.260969 (+132426us) service_pool.cc:225] Handling call
0110 20:57:24.857747 (+5596778us) ts_tablet_manager.cc:1145] Deleting tablet f0b5ba9f38c1493bba01180744f080e1
0110 20:57:24.857800 (+ 53us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.857953 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5588275}
I20250110 20:57:24.860460 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 273) took 5683 ms. Trace:
W20250110 20:57:24.860476 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet f0b5ba9f38c1493bba01180744f080e1 because the tablet was not found. No further retry: Not found: Tablet not found: f0b5ba9f38c1493bba01180744f080e1
I20250110 20:57:24.861016 5594 tablet_bootstrap.cc:654] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:24.860742 5592 rpcz_store.cc:276] 0110 20:57:19.177477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.270526 (+ 93049us) service_pool.cc:225] Handling call
0110 20:57:24.860183 (+5589657us) ts_tablet_manager.cc:1145] Deleting tablet e27e43461251413b97721323cd935374
0110 20:57:24.860254 (+ 71us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.860416 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5589167}
W20250110 20:57:24.861948 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet e27e43461251413b97721323cd935374 because the tablet was not found. No further retry: Not found: Tablet not found: e27e43461251413b97721323cd935374
I20250110 20:57:24.862533 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 274) took 5684 ms. Trace:
I20250110 20:57:24.862807 5592 rpcz_store.cc:276] 0110 20:57:19.177790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.271081 (+ 93291us) service_pool.cc:225] Handling call
0110 20:57:24.862285 (+5591204us) ts_tablet_manager.cc:1145] Deleting tablet d1afe41a09954fc8873ea53a330ea03f
0110 20:57:24.862328 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.862490 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5590754}
W20250110 20:57:24.863639 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet d1afe41a09954fc8873ea53a330ea03f because the tablet was not found. No further retry: Not found: Tablet not found: d1afe41a09954fc8873ea53a330ea03f
I20250110 20:57:24.866384 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40232 (request call id 271) took 5730 ms. Trace:
I20250110 20:57:24.866669 5592 rpcz_store.cc:276] 0110 20:57:19.135813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:19.267748 (+131935us) service_pool.cc:225] Handling call
0110 20:57:24.863516 (+5595768us) ts_tablet_manager.cc:1145] Deleting tablet 19d6eca349514b9fb0fff7cd6230027c
0110 20:57:24.863568 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:57:24.866338 (+ 2770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5559445}
I20250110 20:57:24.867523 5592 tablet_replica.cc:331] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: stopping tablet replica
W20250110 20:57:24.867806 4885 catalog_manager.cc:4839] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): delete failed for tablet 19d6eca349514b9fb0fff7cd6230027c because the tablet was not found. No further retry: Not found: Tablet not found: 19d6eca349514b9fb0fff7cd6230027c
I20250110 20:57:24.868216 5592 raft_consensus.cc:2238] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 LEADER]: Raft consensus shutting down.
W20250110 20:57:24.861146 5399 consensus_peers.cc:487] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:24.883471 5592 pending_rounds.cc:62] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Trying to abort 1 pending ops.
I20250110 20:57:24.883677 5592 pending_rounds.cc:69] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878669294338048 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "da6a9a054aa54455b4979e4c633ad6b4" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 2 new_table_name: "default.Table_853611a0967d473eab31c83128ebe9fc" dest_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" new_extra_config { } }
W20250110 20:57:24.884507 5592 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:40232: Aborted: Op aborted
I20250110 20:57:24.884891 5592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:40232 (request call id 337) took 1015 ms. Trace:
I20250110 20:57:24.885115 5592 rpcz_store.cc:276] 0110 20:57:23.869265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:23.869390 (+ 125us) service_pool.cc:225] Handling call
0110 20:57:24.884844 (+1015454us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:23.870314 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0110 20:57:23.870602 (+ 288us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0110 20:57:23.870628 (+ 26us) alter_schema_op.cc:66] Acquired schema lock
0110 20:57:23.870683 (+ 55us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0110 20:57:23.870778 (+ 95us) alter_schema_op.cc:128] START. Timestamp: 1736542643870688
0110 20:57:23.870795 (+ 17us) op_driver.cc:348] REPLICATION: starting
0110 20:57:23.871249 (+ 454us) log.cc:844] Serialized 173 byte log entry
0110 20:57:24.884396 (+1013147us) op_driver.cc:464] REPLICATION: finished
0110 20:57:24.884425 (+ 29us) op_driver.cc:389] HandleFailure(Aborted: Op aborted)
0110 20:57:24.884443 (+ 18us) alter_schema_op.cc:171] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"mutex_wait_us":784,"prepare.queue_time_us":105,"prepare.run_cpu_time_us":12846,"prepare.run_wall_time_us":150403,"raft.queue_time_us":26804,"raft.run_cpu_time_us":659,"raft.run_wall_time_us":658,"replication_time_us":1013499,"thread_start_us":116269,"threads_started":1,"wal-append.queue_time_us":186306}]]}
W20250110 20:57:24.887260 4885 catalog_manager.cc:4951] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843): alter failed for tablet da6a9a054aa54455b4979e4c633ad6b4 (table default.Table_853611a0967d473eab31c83128ebe9fc [id=b65f8d62ac7f44169142988e1c4903d9]): Aborted: Op aborted
I20250110 20:57:24.887806 5592 raft_consensus.cc:2267] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:24.889901 5592 ts_tablet_manager.cc:1905] T da6a9a054aa54455b4979e4c633ad6b4 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:24.893906 5399 consensus_peers.cc:487] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.894277 5399 consensus_peers.cc:487] T a7a09ea54e79485b9892d52d0e04da85 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.894603 5399 consensus_peers.cc:487] T f2cbb46ea11c4e7d8bc5993ccdc52247 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.894912 5399 consensus_peers.cc:487] T 7309d13f62524b6ab8b4c551728ecb6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.895222 5399 consensus_peers.cc:487] T 57ef3773a87e4a23a19ac1e5f603d2ce P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.895519 5399 consensus_peers.cc:487] T 0af278622ef6432fb0891cb06bc6cd6d P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.895802 5399 leader_election.cc:336] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111)
W20250110 20:57:24.896313 5399 consensus_peers.cc:487] T eee23ab7ecce465ca1153d22c40bf935 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.896618 5399 leader_election.cc:336] T 758430b311a8478c9cab2bcf79253b52 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111)
W20250110 20:57:24.897114 5399 leader_election.cc:336] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111)
I20250110 20:57:24.903347 5594 tablet_bootstrap.cc:492] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.903719 5594 ts_tablet_manager.cc:1397] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.005s
I20250110 20:57:24.905843 5594 raft_consensus.cc:357] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.906452 5594 raft_consensus.cc:383] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.906690 5594 raft_consensus.cc:738] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.907207 5594 consensus_queue.cc:260] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.916047 5594 ts_tablet_manager.cc:1428] T f482cd146ca34f9e95b2361e9c0a985a P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20250110 20:57:24.916837 5594 tablet_bootstrap.cc:492] T 3fdb2ee9419244a9ab939be56b2d7c72 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Bootstrap starting.
I20250110 20:57:24.917479 5399 leader_election.cc:304] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3c5bff1e70804ab4a1bb63bdb9345ce6; no voters: 46005a5920cc4a3697a5c61b1c271384, ac9003be04f340f3886b5625cc928eb0
I20250110 20:57:24.918130 6356 raft_consensus.cc:2743] T 7df3aafea18f49249cceb8a9431064e6 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:24.918296 5399 consensus_peers.cc:487] T 432078ac2fe347e1a55834ca0f77531a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063): Couldn't send request to peer ac9003be04f340f3886b5625cc928eb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.193:36063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:24.924685 4900 catalog_manager.cc:5618] TS 3c5bff1e70804ab4a1bb63bdb9345ce6 (127.4.171.195:32843) does not have the latest schema for tablet f80685cb3c494283bea421195bf7751f (table default.Table_3f641bf1cdfd46fda1bd487fa4f01f1e [id=75dcee4b73484274bac56a2ab1a827d7]). Expected version 1 got 0
I20250110 20:57:24.927702 5428 tablet_service.cc:1514] Processing DeleteTablet for tablet a687769f69d844cc8564e61dc324d2b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:12 UTC) from {username='slave'} at 127.0.0.1:40232
I20250110 20:57:24.934487 5594 tablet_bootstrap.cc:654] T 3fdb2ee9419244a9ab939be56b2d7c72 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:24.937691 5398 connection.cc:537] client connection to 127.4.171.194:40295 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:24.946250 6556 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:57:24.939910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.194:40295 (local address 127.4.171.195:52295)
0110 20:57:24.940990 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0110 20:57:24.941035 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0110 20:57:24.941378 (+ 343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:57:24.945530 (+ 4152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.171.194:40295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":513}
W20250110 20:57:24.946959 5398 consensus_peers.cc:487] T 2d5c46de95bc4227bffc5e8425b8cc95 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.194:40295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.947369 5398 consensus_peers.cc:487] T 371fb7408f0149c8b00659e09ebaf48a P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.194:40295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.947710 5398 consensus_peers.cc:487] T 5eb75e1886e74a719dc09ccad8e1e284 P 3c5bff1e70804ab4a1bb63bdb9345ce6 -> Peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Couldn't send request to peer 46005a5920cc4a3697a5c61b1c271384. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.194:40295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20250110 20:57:24.949368 6557 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:57:24.943729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.194:40295 (local address 127.0.0.1:59526)
0110 20:57:24.944639 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0110 20:57:24.944680 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0110 20:57:24.945012 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:57:24.948546 (+ 3534us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.171.194:40295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":369}
I20250110 20:57:24.951725 6205 raft_consensus.cc:491] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:24.952114 6205 raft_consensus.cc:513] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } }
I20250110 20:57:24.954165 6205 leader_election.cc:290] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac9003be04f340f3886b5625cc928eb0 (127.4.171.193:36063), 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295)
W20250110 20:57:24.961536 5398 leader_election.cc:336] T 33db8a2fdd9c4fde985345604fd04215 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer 46005a5920cc4a3697a5c61b1c271384 (127.4.171.194:40295): Network error: Client connection negotiation failed: client connection to 127.4.171.194:40295: connect: Connection refused (error 111)
I20250110 20:57:24.963634 5594 tablet_bootstrap.cc:492] T 3fdb2ee9419244a9ab939be56b2d7c72 P 3c5bff1e70804ab4a1bb63bdb9345ce6: No bootstrap required, opened a new log
I20250110 20:57:24.964020 5594 ts_tablet_manager.cc:1397] T 3fdb2ee9419244a9ab939be56b2d7c72 P 3c5bff1e70804ab4a1bb63bdb9345ce6: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.000s
I20250110 20:57:24.967032 5594 raft_consensus.cc:357] T 3fdb2ee9419244a9ab939be56b2d7c72 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac9003be04f340f3886b5625cc928eb0" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36063 } } peers { permanent_uuid: "3c5bff1e70804ab4a1bb63bdb9345ce6" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 32843 } } peers { permanent_uuid: "46005a5920cc4a3697a5c61b1c271384" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 40295 } }
I20250110 20:57:24.967684 5594 raft_consensus.cc:383] T 3fdb2ee9419244a9ab939be56b2d7c72 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:24.967950 5594 raft_consensus.cc:738] T 3fdb2ee9419244a9ab939be56b2d7c72 P 3c5bff1e70804ab4a1bb63bdb9345ce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5bff1e70804ab4a1bb63bdb9345ce6, State: Initialized, Role: FOLLOWER
I20250110 20:57:24.968149 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 5378
W20250110 20:57:25.107275 6009 connection.cc:537] server connection from 127.4.171.195:46805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:25.107388 4809 connection.cc:537] server connection from 127.4.171.195:57321 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:25.108139 4885 connection.cc:537] client connection to 127.4.171.195:32843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250110 20:57:25.118746 6569 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:57:25.115627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.195:32843 (local address 127.0.0.1:51100)
0110 20:57:25.116588 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0110 20:57:25.116627 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0110 20:57:25.116867 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:57:25.118131 (+ 1264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.171.195:32843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":443}
I20250110 20:57:25.125569 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 4792
I20250110 20:57:25.195326 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 4862
W20250110 20:57:25.195320 4887 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84772eba3f0460faf8523f76eb57e53 -> Peer 63aa77ee6ec84a6e82132b2de9e77486 (127.4.171.254:34617): Couldn't send request to peer 63aa77ee6ec84a6e82132b2de9e77486. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.254:34617: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:25.269431 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 5581
2025-01-10T20:57:25Z chronyd exiting
I20250110 20:57:25.314626 4783 master-stress-test.cc:585] Tables created: 162
I20250110 20:57:25.314801 4783 master-stress-test.cc:586] Tables altered: 77
I20250110 20:57:25.314968 4783 master-stress-test.cc:587] Tables deleted: 61
I20250110 20:57:25.315107 4783 master-stress-test.cc:588] Tablets replaced: 34
I20250110 20:57:25.315255 4783 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (36489 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
I20250110 20:57:25.380506 4783 test_util.cc:274] Using random seed: 1468261763
2025-01-10T20:57:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-01-10T20:57:25Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1Eq_p2/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1Eq_p2/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/security.properties
2025-01-10 20:57:31: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1Eq_p2/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-task1Eq_p2/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20250110 20:57:39.328766 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.254:36415
--webserver_interface=127.4.171.254
--webserver_port=0
--builtin_ntp_servers=127.4.171.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--hive_metastore_uris=thrift://127.0.0.1:43427
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250110 20:57:39.615782 6808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:39.616362 6808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:39.616802 6808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:57:39.646996 6808 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250110 20:57:39.647256 6808 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250110 20:57:39.647501 6808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:57:39.647747 6808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:39.647980 6808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:57:39.648206 6808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:57:39.682264 6808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:43427
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.254:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.171.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:39.683569 6808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:39.685210 6808 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:57:39.699914 6816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:57:39.699916 6815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:39.701870 6808 server_base.cc:1034] running on GCE node
W20250110 20:57:39.701238 6818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:40.775549 6808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:40.778386 6808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:40.779788 6808 hybrid_clock.cc:648] HybridClock initialized: now 1736542660779750 us; error 54 us; skew 500 ppm
I20250110 20:57:40.780568 6808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:40.786710 6808 webserver.cc:458] Webserver started at http://127.4.171.254:38411/ using document root <none> and password file <none>
I20250110 20:57:40.787587 6808 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:40.787801 6808 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:57:40.788240 6808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:57:40.794262 6808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data/instance:
uuid: "5496a4fb46f84c47a170dfe492f14b51"
format_stamp: "Formatted at 2025-01-10 20:57:40 on dist-test-slave-98kh"
I20250110 20:57:40.795303 6808 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/wal/instance:
uuid: "5496a4fb46f84c47a170dfe492f14b51"
format_stamp: "Formatted at 2025-01-10 20:57:40 on dist-test-slave-98kh"
I20250110 20:57:40.802047 6808 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250110 20:57:40.807107 6825 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:40.808079 6808 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250110 20:57:40.808384 6808 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/wal
uuid: "5496a4fb46f84c47a170dfe492f14b51"
format_stamp: "Formatted at 2025-01-10 20:57:40 on dist-test-slave-98kh"
I20250110 20:57:40.808696 6808 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:40.869289 6808 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:40.870744 6808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:40.871165 6808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:40.935061 6808 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.254:36415
I20250110 20:57:40.935134 6876 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.254:36415 every 8 connection(s)
I20250110 20:57:40.937692 6808 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/data/info.pb
I20250110 20:57:40.943035 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 6808
I20250110 20:57:40.943415 6877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:40.943648 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-0/wal/instance
I20250110 20:57:40.947193 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.253:40203
--webserver_interface=127.4.171.253
--webserver_port=0
--builtin_ntp_servers=127.4.171.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--hive_metastore_uris=thrift://127.0.0.1:43427
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 20:57:40.954450 6877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:57:40.977532 6877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } has no permanent_uuid. Determining permanent_uuid...
W20250110 20:57:40.979730 6828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
W20250110 20:57:40.983384 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:40203: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:41.032800 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } attempt: 1
W20250110 20:57:41.036396 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:40203: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:41.104732 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } attempt: 2
W20250110 20:57:41.108305 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:40203: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:41.199643 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } attempt: 3
W20250110 20:57:41.203164 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:40203: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
W20250110 20:57:41.241420 6878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:41.241992 6878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:41.242429 6878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:57:41.272303 6878 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250110 20:57:41.272588 6878 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250110 20:57:41.272846 6878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:57:41.273094 6878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:41.273337 6878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:57:41.273579 6878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:57:41.309551 6878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43427
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.253:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.171.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:41.310845 6878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:41.312335 6878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:57:41.326823 6887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:41.346484 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } attempt: 4
W20250110 20:57:41.350375 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:40203: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:41.649745 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } attempt: 5
W20250110 20:57:41.653601 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:40203: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:42.200979 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } attempt: 6
W20250110 20:57:42.207463 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.253:40203: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
W20250110 20:57:41.327025 6888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:57:41.327713 6890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:41.328349 6878 server_base.cc:1034] running on GCE node
I20250110 20:57:42.415179 6878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:42.417637 6878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:42.418965 6878 hybrid_clock.cc:648] HybridClock initialized: now 1736542662418943 us; error 40 us; skew 500 ppm
I20250110 20:57:42.419752 6878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:42.425524 6878 webserver.cc:458] Webserver started at http://127.4.171.253:33297/ using document root <none> and password file <none>
I20250110 20:57:42.426399 6878 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:42.426606 6878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:57:42.427044 6878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:57:42.431887 6878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/instance:
uuid: "f9bb61bdf30044a1b482d6a66e29435f"
format_stamp: "Formatted at 2025-01-10 20:57:42 on dist-test-slave-98kh"
I20250110 20:57:42.432929 6878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal/instance:
uuid: "f9bb61bdf30044a1b482d6a66e29435f"
format_stamp: "Formatted at 2025-01-10 20:57:42 on dist-test-slave-98kh"
I20250110 20:57:42.439796 6878 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20250110 20:57:42.445021 6899 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:42.446036 6878 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250110 20:57:42.446334 6878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
uuid: "f9bb61bdf30044a1b482d6a66e29435f"
format_stamp: "Formatted at 2025-01-10 20:57:42 on dist-test-slave-98kh"
I20250110 20:57:42.446650 6878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:42.508414 6878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:42.509862 6878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:42.510267 6878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:42.574762 6878 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.253:40203
I20250110 20:57:42.574865 6950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.253:40203 every 8 connection(s)
I20250110 20:57:42.577458 6878 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
I20250110 20:57:42.582108 6951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:42.582808 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 6878
I20250110 20:57:42.583174 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal/instance
I20250110 20:57:42.585532 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.252:33621
--webserver_interface=127.4.171.252
--webserver_port=0
--builtin_ntp_servers=127.4.171.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--hive_metastore_uris=thrift://127.0.0.1:43427
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 20:57:42.592592 6951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:57:42.612098 6951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:57:42.625759 6951 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } has no permanent_uuid. Determining permanent_uuid...
W20250110 20:57:42.627821 6902 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
W20250110 20:57:42.631541 6951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:42.680995 6951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 1
W20250110 20:57:42.684554 6951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:42.753016 6951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 2
W20250110 20:57:42.756632 6951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:42.848078 6951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 3
W20250110 20:57:42.851719 6951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
W20250110 20:57:42.878623 6952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:42.879127 6952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:42.879511 6952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:57:42.909026 6952 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250110 20:57:42.909240 6952 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250110 20:57:42.909451 6952 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:57:42.909644 6952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:42.909837 6952 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:57:42.910028 6952 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:57:42.953584 6952 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:43427
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.252:33621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.171.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:42.955286 6952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:42.957296 6952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:57:42.972446 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:42.995138 6951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 4
W20250110 20:57:42.998893 6951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.268050 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } attempt: 7
I20250110 20:57:43.290830 6877 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } has no permanent_uuid. Determining permanent_uuid...
W20250110 20:57:43.295972 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.298236 6951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 5
W20250110 20:57:43.301752 6951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.354346 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 1
W20250110 20:57:43.358762 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.440140 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 2
W20250110 20:57:43.444306 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.529661 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 3
W20250110 20:57:43.533881 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.674242 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 4
W20250110 20:57:43.678604 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.849141 6951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 6
W20250110 20:57:43.855547 6951 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
I20250110 20:57:43.961987 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 5
W20250110 20:57:43.965916 6877 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.171.252:33621: Network error: Client connection negotiation failed: client connection to 127.4.171.252:33621: connect: Connection refused (error 111)
W20250110 20:57:42.973513 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:42.974361 6952 server_base.cc:1034] running on GCE node
W20250110 20:57:42.972457 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:44.069525 6952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:44.072376 6952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:44.073777 6952 hybrid_clock.cc:648] HybridClock initialized: now 1736542664073748 us; error 42 us; skew 500 ppm
I20250110 20:57:44.074520 6952 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:44.080672 6952 webserver.cc:458] Webserver started at http://127.4.171.252:43079/ using document root <none> and password file <none>
I20250110 20:57:44.081573 6952 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:44.081776 6952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:57:44.082170 6952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:57:44.086946 6952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data/instance:
uuid: "a36f292a1a20461e9aef5aefe8fd0f08"
format_stamp: "Formatted at 2025-01-10 20:57:44 on dist-test-slave-98kh"
I20250110 20:57:44.087975 6952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/wal/instance:
uuid: "a36f292a1a20461e9aef5aefe8fd0f08"
format_stamp: "Formatted at 2025-01-10 20:57:44 on dist-test-slave-98kh"
I20250110 20:57:44.094765 6952 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250110 20:57:44.099979 6977 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:44.100955 6952 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250110 20:57:44.101245 6952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/wal
uuid: "a36f292a1a20461e9aef5aefe8fd0f08"
format_stamp: "Formatted at 2025-01-10 20:57:44 on dist-test-slave-98kh"
I20250110 20:57:44.101584 6952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:44.151299 6952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:44.152670 6952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:44.153092 6952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:44.217509 6952 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.252:33621
I20250110 20:57:44.217584 7028 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.252:33621 every 8 connection(s)
I20250110 20:57:44.220103 6952 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/data/info.pb
I20250110 20:57:44.222270 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 6952
I20250110 20:57:44.222637 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-2/wal/instance
I20250110 20:57:44.224751 7029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:44.241731 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:57:44.262452 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:57:44.271647 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } has no permanent_uuid. Determining permanent_uuid...
I20250110 20:57:44.287772 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Bootstrap starting.
I20250110 20:57:44.293474 7029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:44.295101 7029 log.cc:826] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Log is configured to *not* fsync() on all Append() calls
I20250110 20:57:44.299518 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: No bootstrap required, opened a new log
I20250110 20:57:44.320438 7029 raft_consensus.cc:357] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:44.321358 7029 raft_consensus.cc:383] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:44.321719 7029 raft_consensus.cc:738] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36f292a1a20461e9aef5aefe8fd0f08, State: Initialized, Role: FOLLOWER
I20250110 20:57:44.322502 7029 consensus_queue.cc:260] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:44.324635 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:44.325243 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:44.325984 7029 sys_catalog.cc:564] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: configured and running, proceeding with master startup.
W20250110 20:57:44.356405 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.356770 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.379235 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.379663 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.353643 6952 master_runner.cc:186] Error getting master registration for 127.4.171.254:36415: OK, instance_id { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" instance_seqno: 1736542660917712 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250110 20:57:44.386402 6952 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250110 20:57:44.393756 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.394017 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.407804 6983 tablet.cc:2367] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:57:44.408092 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.408357 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.421545 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.421780 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.434852 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.435093 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.447870 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.448108 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.462179 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.462416 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.475396 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.475652 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.488523 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.488775 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.503037 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.503397 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.516271 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.516498 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.518280 6877 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 6
W20250110 20:57:44.531912 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.532222 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.536602 6877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Bootstrap starting.
I20250110 20:57:44.541867 6877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:44.543463 6877 log.cc:826] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Log is configured to *not* fsync() on all Append() calls
W20250110 20:57:44.546550 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.546926 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.547889 6877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: No bootstrap required, opened a new log
W20250110 20:57:44.559811 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.560045 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.574846 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.575179 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.579164 6877 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:44.580063 6877 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:44.580394 6877 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5496a4fb46f84c47a170dfe492f14b51, State: Initialized, Role: FOLLOWER
I20250110 20:57:44.581261 6877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:44.583952 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:44.584811 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:44.585829 6877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: configured and running, proceeding with master startup.
W20250110 20:57:44.589336 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.589679 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.603835 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.604163 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.618589 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.619052 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.629736 6831 tablet.cc:2367] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:57:44.636948 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.637284 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.640558 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.640893 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.651423 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.651788 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.655186 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.655480 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.627826 6808 master_runner.cc:186] Error getting master registration for 127.4.171.253:40203: OK, instance_id { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" instance_seqno: 1736542662557120 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250110 20:57:44.663825 6808 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250110 20:57:44.665926 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.666208 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.670271 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.670579 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.680411 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.680714 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.684751 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.685039 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.694869 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.695178 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.699185 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.700455 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.709285 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.709606 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.714661 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.714956 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.723740 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.724057 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.729218 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.729545 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.738890 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.739255 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.743670 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.743966 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.751845 7036 raft_consensus.cc:491] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:44.752367 7036 raft_consensus.cc:513] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
W20250110 20:57:44.753501 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.753769 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.754824 7036 leader_election.cc:290] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203)
I20250110 20:57:44.757301 6852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:57:44.758211 6852 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 0.
W20250110 20:57:44.758098 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.759348 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.759709 6980 leader_election.cc:304] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08; no voters:
I20250110 20:57:44.760576 7036 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:44.760898 7036 raft_consensus.cc:491] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:44.761224 7036 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:44.767576 7036 raft_consensus.cc:513] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:44.769385 7036 leader_election.cc:290] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 1 election: Requested vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203)
W20250110 20:57:44.773773 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250110 20:57:44.773864 6852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51"
I20250110 20:57:44.774427 6852 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:44.777832 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.778100 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.774088 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.785531 6852 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 1.
I20250110 20:57:44.787142 6980 leader_election.cc:304] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08; no voters:
I20250110 20:57:44.788002 7036 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:44.795498 7036 raft_consensus.cc:695] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 1 LEADER]: Becoming Leader. State: Replica: a36f292a1a20461e9aef5aefe8fd0f08, State: Running, Role: LEADER
I20250110 20:57:44.796447 7036 consensus_queue.cc:237] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
W20250110 20:57:44.802292 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.802978 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.822901 7072 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 1 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:44.825335 7072 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
W20250110 20:57:44.835271 7050 catalog_manager.cc:1559] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.835572 7050 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:44.840286 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250110 20:57:44.842536 7074 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
W20250110 20:57:44.842442 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.849758 7074 catalog_manager.cc:1485] Initializing Kudu cluster ID...
W20250110 20:57:44.861179 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.861500 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.872983 6852 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Refusing update from remote peer a36f292a1a20461e9aef5aefe8fd0f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250110 20:57:44.875533 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20250110 20:57:44.875315 7036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:57:44.875825 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.912350 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 1 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:44.913100 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:44.917032 6951 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } attempt: 7
W20250110 20:57:44.921850 7070 catalog_manager.cc:1559] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250110 20:57:44.922343 7070 catalog_manager.cc:874] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.937644 7075 mvcc.cc:204] Tried to move back new op lower bound from 7112878755302584320 to 7112878755018166272. Current Snapshot: MvccSnapshot[applied={T|T < 7112878755302584320}]
I20250110 20:57:44.955600 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:44.956354 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:44.956843 7076 mvcc.cc:204] Tried to move back new op lower bound from 7112878755302584320 to 7112878755018166272. Current Snapshot: MvccSnapshot[applied={T|T < 7112878755302584320}]
I20250110 20:57:44.962102 7055 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:44.964043 7055 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:44.968240 7070 catalog_manager.cc:1260] Loaded cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:57:44.968472 7070 catalog_manager.cc:1553] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: loading cluster ID for follower catalog manager: success
I20250110 20:57:44.975934 7074 catalog_manager.cc:1348] Generated new cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:57:44.976235 7074 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:57:44.983917 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: Bootstrap starting.
W20250110 20:57:44.992326 7070 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250110 20:57:44.992635 7070 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:44.999650 6951 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:45.013159 6951 log.cc:826] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: Log is configured to *not* fsync() on all Append() calls
W20250110 20:57:45.014895 7070 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250110 20:57:45.015203 7070 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:45.032164 7070 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250110 20:57:45.032514 7070 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
W20250110 20:57:45.058100 7070 catalog_manager.cc:1577] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20250110 20:57:45.058410 7070 catalog_manager.cc:874] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20250110 20:57:45.069540 7074 catalog_manager.cc:1371] Generated new certificate authority record
I20250110 20:57:45.072204 6951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: No bootstrap required, opened a new log
I20250110 20:57:45.074268 7074 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:57:45.098029 7070 catalog_manager.cc:1575] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: acquiring CA information for follower catalog manager: success
W20250110 20:57:45.109912 7070 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20250110 20:57:45.148874 7070 catalog_manager.cc:874] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250110 20:57:45.153667 7074 catalog_manager.cc:5899] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Generated new TSK 0
I20250110 20:57:45.154770 7074 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:57:45.156014 6951 raft_consensus.cc:357] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:45.156965 6951 raft_consensus.cc:383] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:45.157303 6951 raft_consensus.cc:738] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bb61bdf30044a1b482d6a66e29435f, State: Initialized, Role: FOLLOWER
I20250110 20:57:45.158279 6951 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:45.167918 7070 catalog_manager.cc:1603] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 20:57:45.167718 7074 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250110 20:57:45.170378 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:45.171187 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:45.172763 6951 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:57:45.172950 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:45.173806 6926 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 0.
I20250110 20:57:45.182348 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f"
I20250110 20:57:45.183005 6925 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:45.192971 6925 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 1.
I20250110 20:57:45.194288 6924 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 1 FOLLOWER]: Refusing update from remote peer a36f292a1a20461e9aef5aefe8fd0f08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:45.201617 7036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:57:45.211066 6925 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:51586 (request call id 2) took 438 ms (client timeout 534 ms). Trace:
W20250110 20:57:45.233204 6925 rpcz_store.cc:269] 0110 20:57:44.772686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:44.772828 (+ 142us) service_pool.cc:225] Handling call
0110 20:57:45.210989 (+438161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:57:45.241573 7074 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250110 20:57:45.286572 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.171.193:0
--local_ip_for_outbound_sockets=127.4.171.193
--webserver_interface=127.4.171.193
--webserver_port=0
--tserver_master_addrs=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--builtin_ntp_servers=127.4.171.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 20:57:45.304435 7082 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 1 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:45.305192 7082 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:45.304533 7087 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:45.305647 7087 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:45.380270 7103 catalog_manager.cc:796] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250110 20:57:45.432917 7103 catalog_manager.cc:1260] Loaded cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:57:45.433182 7103 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: loading cluster ID for follower catalog manager: success
I20250110 20:57:45.444002 7103 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: acquiring CA information for follower catalog manager: success
I20250110 20:57:45.450573 7103 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 20:57:45.632930 7036 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:45.633750 7036 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:45.679533 7105 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:45.683681 7105 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:45.696676 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:45.699715 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:45.701352 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51; no voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f
I20250110 20:57:45.706449 7105 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:45.707582 7105 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250110 20:57:45.874341 7093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:45.875144 7093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:45.876143 7093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:57:45.933692 7093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:45.935290 7093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.171.193
I20250110 20:57:45.998018 7093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.171.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.171.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.171.193
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:45.999984 7093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:46.002147 7093 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:57:46.043841 7113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:46.554482 7118 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:46.555163 7118 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:46.557575 7118 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:46.558943 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:57:46.560209 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:46.562245 6902 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:57:46.563084 7118 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250110 20:57:47.114722 7119 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:47.115434 7119 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.118170 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:47.118760 6926 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5496a4fb46f84c47a170dfe492f14b51 in term 1.
I20250110 20:57:47.120028 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, f9bb61bdf30044a1b482d6a66e29435f; no voters:
I20250110 20:57:47.130733 7119 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:47.131995 7119 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:57:47.132802 7119 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:47.133579 7119 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:47.140990 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:47.146646 7119 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.163483 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f"
I20250110 20:57:47.164065 6926 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:47.171452 6926 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5496a4fb46f84c47a170dfe492f14b51 in term 2.
I20250110 20:57:47.172986 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, f9bb61bdf30044a1b482d6a66e29435f; no voters:
I20250110 20:57:47.174021 7121 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:57:47.178872 7119 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 2 election: Requested vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:47.179100 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08"
I20250110 20:57:47.181198 6926 raft_consensus.cc:1235] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 2 FOLLOWER]: Rejecting Update request from peer a36f292a1a20461e9aef5aefe8fd0f08 for earlier term 1. Current term is 2. Ops: []
I20250110 20:57:47.182667 7120 consensus_queue.cc:1046] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 20:57:47.186385 7121 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 2 LEADER]: Becoming Leader. State: Replica: 5496a4fb46f84c47a170dfe492f14b51, State: Running, Role: LEADER
I20250110 20:57:47.187414 7121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.197199 7119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5496a4fb46f84c47a170dfe492f14b51. Latest consensus state: current_term: 2 leader_uuid: "5496a4fb46f84c47a170dfe492f14b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:47.197937 7119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:47.202525 7125 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:57:47.219310 7123 raft_consensus.cc:3049] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 1 LEADER]: Stepping down as leader of term 1
I20250110 20:57:47.225502 7123 raft_consensus.cc:738] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a36f292a1a20461e9aef5aefe8fd0f08, State: Running, Role: LEADER
I20250110 20:57:47.226518 7123 consensus_queue.cc:260] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.227869 7123 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:47.247586 7050 catalog_manager.cc:1603] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 20:57:47.258983 7125 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:57:47.263773 7125 catalog_manager.cc:1260] Loaded cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:57:47.277530 7125 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:57:47.283423 7125 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:57:47.295372 7125 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Loaded TSK: 0
I20250110 20:57:47.310444 7125 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:57:47.315769 7125 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250110 20:57:47.327214 7125 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250110 20:57:47.691231 7123 raft_consensus.cc:491] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:47.691663 7123 raft_consensus.cc:513] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.693432 7123 leader_election.cc:290] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203)
I20250110 20:57:47.695676 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:57:47.696979 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:47.697540 6926 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 2.
I20250110 20:57:47.699187 6980 leader_election.cc:304] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51
I20250110 20:57:47.700251 7123 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250110 20:57:47.700546 7123 raft_consensus.cc:491] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:47.700839 7123 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:57:47.707885 7123 raft_consensus.cc:513] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.720474 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51"
I20250110 20:57:47.723870 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f"
I20250110 20:57:47.724396 6926 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:57:47.731940 6926 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 3.
I20250110 20:57:47.733234 6980 leader_election.cc:304] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51
I20250110 20:57:47.739049 7123 leader_election.cc:290] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 3 election: Requested vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203)
I20250110 20:57:47.740331 7126 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 3 FOLLOWER]: Leader election won for term 3
I20250110 20:57:47.744259 7126 raft_consensus.cc:695] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 3 LEADER]: Becoming Leader. State: Replica: a36f292a1a20461e9aef5aefe8fd0f08, State: Running, Role: LEADER
I20250110 20:57:47.745043 7126 consensus_queue.cc:237] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.755451 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 3 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:47.756146 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:47.762763 7128 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:57:47.781140 7128 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:57:47.781545 7004 raft_consensus.cc:1235] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 3 LEADER]: Rejecting Update request from peer 5496a4fb46f84c47a170dfe492f14b51 for earlier term 2. Current term is 3. Ops: []
I20250110 20:57:47.789155 7129 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 20:57:47.791674 7129 consensus_queue.cc:1225] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [LEADER]: Peer a36f292a1a20461e9aef5aefe8fd0f08 log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250110 20:57:47.795706 7128 catalog_manager.cc:1260] Loaded cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:57:47.795997 7128 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:57:47.807384 7128 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:57:47.808996 7130 raft_consensus.cc:3049] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 2 LEADER]: Stepping down as leader of term 2
I20250110 20:57:47.812314 7128 catalog_manager.cc:5910] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Loaded TSK: 0
I20250110 20:57:47.815353 6926 raft_consensus.cc:1235] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Rejecting Update request from peer 5496a4fb46f84c47a170dfe492f14b51 for earlier term 2. Current term is 3. Ops: []
I20250110 20:57:47.816926 7129 consensus_queue.cc:1046] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 20:57:47.817525 7128 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:57:47.819458 7130 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5496a4fb46f84c47a170dfe492f14b51, State: Running, Role: LEADER
I20250110 20:57:47.820829 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:47.822049 7128 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250110 20:57:47.822094 7130 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 2 FOLLOWER]: Advancing to term 3
I20250110 20:57:47.833153 7128 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
I20250110 20:57:48.147320 6851 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Refusing update from remote peer a36f292a1a20461e9aef5aefe8fd0f08: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20250110 20:57:48.147675 6851 pending_rounds.cc:77] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Aborting all ops after (but not including) 4
I20250110 20:57:48.147943 6851 pending_rounds.cc:99] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20250110 20:57:48.148180 6851 raft_consensus.cc:2883] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250110 20:57:48.149895 7123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 20:57:48.154414 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 3 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:48.155130 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:48.179291 7130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:48.180020 7130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:48.187124 6926 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Refusing update from remote peer a36f292a1a20461e9aef5aefe8fd0f08: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20250110 20:57:48.188678 7123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250110 20:57:48.198381 7123 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:48.199138 7123 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
W20250110 20:57:46.045850 7112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:57:46.058097 7116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:57:48.224529 7114 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2178 milliseconds
I20250110 20:57:48.224632 7093 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250110 20:57:48.226301 7093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:48.226421 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:48.227351 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:48.224264 7134 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 3 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:48.228264 7134 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:48.229127 7093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:48.230598 7093 hybrid_clock.cc:648] HybridClock initialized: now 1736542668230579 us; error 108 us; skew 500 ppm
I20250110 20:57:48.231731 7093 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:48.239514 7093 webserver.cc:458] Webserver started at http://127.4.171.193:37729/ using document root <none> and password file <none>
I20250110 20:57:48.240931 7093 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:48.241231 7093 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:57:48.241876 7093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:57:48.248912 7093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data/instance:
uuid: "244d9ba396f549f2a5015792b6c5fff8"
format_stamp: "Formatted at 2025-01-10 20:57:48 on dist-test-slave-98kh"
I20250110 20:57:48.250427 7093 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/instance:
uuid: "244d9ba396f549f2a5015792b6c5fff8"
format_stamp: "Formatted at 2025-01-10 20:57:48 on dist-test-slave-98kh"
I20250110 20:57:48.259713 7093 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20250110 20:57:48.266999 7142 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:48.268030 7093 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250110 20:57:48.268321 7093 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal
uuid: "244d9ba396f549f2a5015792b6c5fff8"
format_stamp: "Formatted at 2025-01-10 20:57:48 on dist-test-slave-98kh"
I20250110 20:57:48.268630 7093 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:48.343478 7093 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:48.344919 7093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:48.345340 7093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:48.348105 7093 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:57:48.352800 7093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:57:48.353009 7093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:48.353277 7093 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:57:48.353456 7093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250110 20:57:48.510193 7093 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.193:36745
I20250110 20:57:48.510277 7254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.193:36745 every 8 connection(s)
I20250110 20:57:48.513725 7093 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/data/info.pb
I20250110 20:57:48.516314 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 7093
I20250110 20:57:48.516824 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/instance
I20250110 20:57:48.533380 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.171.194:0
--local_ip_for_outbound_sockets=127.4.171.194
--webserver_interface=127.4.171.194
--webserver_port=0
--tserver_master_addrs=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--builtin_ntp_servers=127.4.171.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 20:57:48.569393 7126 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:48.570151 7126 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:48.603443 7255 heartbeater.cc:346] Connected to a master server at 127.4.171.252:33621
I20250110 20:57:48.603933 7255 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:48.605161 7255 heartbeater.cc:510] Master 127.4.171.252:33621 requested a full tablet report, sending...
I20250110 20:57:48.608502 6994 ts_manager.cc:194] Registered new tserver with Master: 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:57:48.612911 7256 heartbeater.cc:346] Connected to a master server at 127.4.171.253:40203
I20250110 20:57:48.613217 7256 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:48.613497 6994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.171.193:58881
I20250110 20:57:48.614055 7256 heartbeater.cc:510] Master 127.4.171.253:40203 requested a full tablet report, sending...
I20250110 20:57:48.616928 6916 ts_manager.cc:194] Registered new tserver with Master: 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:57:48.620750 7260 heartbeater.cc:346] Connected to a master server at 127.4.171.254:36415
I20250110 20:57:48.621057 7260 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:48.621721 7260 heartbeater.cc:510] Master 127.4.171.254:36415 requested a full tablet report, sending...
I20250110 20:57:48.624394 6842 ts_manager.cc:194] Registered new tserver with Master: 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:57:48.660243 7130 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:48.660698 7130 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:48.662513 7130 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:48.663565 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:48.663589 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:48.665130 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51; no voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f
I20250110 20:57:48.665879 7130 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:57:48.872313 7259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:48.872792 7259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:48.873359 7259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:57:48.904804 7259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:48.905718 7259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.171.194
I20250110 20:57:48.941450 7259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.171.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.171.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.171.194
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:48.943179 7259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:48.945123 7259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:57:48.962025 7271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:49.071930 7277 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:49.072283 7277 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:49.073707 7277 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:49.074556 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:57:49.074842 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:49.076120 6902 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:57:49.076654 7277 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:57:49.117853 7255 heartbeater.cc:502] Master 127.4.171.252:33621 was elected leader, sending a full tablet report...
W20250110 20:57:48.963294 7272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:57:48.963773 7274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:48.965103 7259 server_base.cc:1034] running on GCE node
I20250110 20:57:50.054364 7259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:50.056782 7259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:50.058182 7259 hybrid_clock.cc:648] HybridClock initialized: now 1736542670058159 us; error 42 us; skew 500 ppm
I20250110 20:57:50.058929 7259 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:50.065094 7259 webserver.cc:458] Webserver started at http://127.4.171.194:37969/ using document root <none> and password file <none>
I20250110 20:57:50.065965 7259 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:50.066154 7259 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:57:50.066538 7259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:57:50.071229 7259 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data/instance:
uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
format_stamp: "Formatted at 2025-01-10 20:57:50 on dist-test-slave-98kh"
I20250110 20:57:50.072338 7259 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/instance:
uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
format_stamp: "Formatted at 2025-01-10 20:57:50 on dist-test-slave-98kh"
I20250110 20:57:50.079620 7259 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250110 20:57:50.084973 7284 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:50.086110 7259 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250110 20:57:50.086405 7259 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal
uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
format_stamp: "Formatted at 2025-01-10 20:57:50 on dist-test-slave-98kh"
I20250110 20:57:50.086700 7259 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:50.093668 7285 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:50.094297 7285 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:50.096290 7285 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:50.097240 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:50.097590 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:50.098893 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51; no voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f
I20250110 20:57:50.099507 7285 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:57:50.154021 7259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:50.155417 7259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:50.155799 7259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:50.158231 7259 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:57:50.162112 7259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:57:50.162307 7259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:50.162544 7259 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:57:50.162698 7259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:50.295054 7259 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.194:34035
I20250110 20:57:50.295176 7397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.194:34035 every 8 connection(s)
I20250110 20:57:50.297583 7259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/data/info.pb
I20250110 20:57:50.306195 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 7259
I20250110 20:57:50.306803 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/instance
I20250110 20:57:50.317575 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.171.195:0
--local_ip_for_outbound_sockets=127.4.171.195
--webserver_interface=127.4.171.195
--webserver_port=0
--tserver_master_addrs=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--builtin_ntp_servers=127.4.171.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250110 20:57:50.327565 7398 heartbeater.cc:346] Connected to a master server at 127.4.171.252:33621
I20250110 20:57:50.328068 7398 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:50.329313 7398 heartbeater.cc:510] Master 127.4.171.252:33621 requested a full tablet report, sending...
I20250110 20:57:50.331517 7399 heartbeater.cc:346] Connected to a master server at 127.4.171.253:40203
I20250110 20:57:50.331802 7399 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:50.331844 6994 ts_manager.cc:194] Registered new tserver with Master: 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:50.332588 7399 heartbeater.cc:510] Master 127.4.171.253:40203 requested a full tablet report, sending...
I20250110 20:57:50.333492 6994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.171.194:59945
I20250110 20:57:50.337469 6916 ts_manager.cc:194] Registered new tserver with Master: 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:50.340044 7400 heartbeater.cc:346] Connected to a master server at 127.4.171.254:36415
I20250110 20:57:50.340369 7400 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:50.340885 7400 heartbeater.cc:510] Master 127.4.171.254:36415 requested a full tablet report, sending...
I20250110 20:57:50.342800 6842 ts_manager.cc:194] Registered new tserver with Master: 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:50.640053 7413 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:50.640483 7413 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:50.642196 7413 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:50.642833 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:50.643136 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:50.644290 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51; no voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f
I20250110 20:57:50.644946 7413 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250110 20:57:50.650756 7408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:50.651350 7408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:50.651988 7408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250110 20:57:50.682416 7408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:50.683306 7408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.171.195
I20250110 20:57:50.717115 7408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.171.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.171.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.171.195
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:50.718430 7408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:50.719955 7408 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250110 20:57:50.735761 7416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:50.840683 7398 heartbeater.cc:502] Master 127.4.171.252:33621 was elected leader, sending a full tablet report...
I20250110 20:57:51.106508 7421 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:51.106920 7421 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.108294 7421 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:51.109397 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:51.109506 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:57:51.111097 6902 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:57:51.111670 7421 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250110 20:57:51.153304 7422 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:51.153687 7422 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.155128 7422 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:51.156105 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:51.156070 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:51.156746 6926 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5496a4fb46f84c47a170dfe492f14b51 in term 3.
I20250110 20:57:51.157784 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, f9bb61bdf30044a1b482d6a66e29435f; no voters: a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:57:51.158335 7422 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250110 20:57:51.158617 7422 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Starting leader election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:51.158855 7422 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:57:51.163524 7422 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.165841 7422 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 election: Requested vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:51.165956 6926 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f"
I20250110 20:57:51.166879 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08"
I20250110 20:57:51.168555 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51; no voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f
I20250110 20:57:51.169319 7422 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20250110 20:57:51.198999 6851 raft_consensus.cc:1235] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 4 FOLLOWER]: Rejecting Update request from peer a36f292a1a20461e9aef5aefe8fd0f08 for earlier term 3. Current term is 4. Ops: []
I20250110 20:57:51.200353 7423 consensus_queue.cc:1046] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 20:57:51.201673 7424 raft_consensus.cc:3049] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 3 LEADER]: Stepping down as leader of term 3
I20250110 20:57:51.201910 7424 raft_consensus.cc:738] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: a36f292a1a20461e9aef5aefe8fd0f08, State: Running, Role: LEADER
I20250110 20:57:51.202482 7424 consensus_queue.cc:260] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.203351 7424 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 3 FOLLOWER]: Advancing to term 4
I20250110 20:57:51.663651 7424 raft_consensus.cc:491] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:51.663997 7424 raft_consensus.cc:513] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.665684 7424 leader_election.cc:290] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203)
I20250110 20:57:51.666404 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:57:51.666949 6851 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 4.
I20250110 20:57:51.667089 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:57:51.667695 6980 leader_election.cc:304] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08; no voters:
I20250110 20:57:51.667896 6925 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 3.
I20250110 20:57:51.668457 7424 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20250110 20:57:51.668762 7424 raft_consensus.cc:491] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:51.669062 7424 raft_consensus.cc:3054] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 4 FOLLOWER]: Advancing to term 5
I20250110 20:57:51.670202 7426 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:57:51.670666 7426 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.672837 7426 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:57:51.673923 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:57:51.673974 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:57:51.674672 6851 raft_consensus.cc:2388] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9bb61bdf30044a1b482d6a66e29435f in current term 4: Already voted for candidate 5496a4fb46f84c47a170dfe492f14b51 in this term.
I20250110 20:57:51.676558 7424 raft_consensus.cc:513] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.677640 7004 raft_consensus.cc:2363] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9bb61bdf30044a1b482d6a66e29435f for earlier term 4. Current term is 5.
I20250110 20:57:51.678575 7424 leader_election.cc:290] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 5 election: Requested vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203)
I20250110 20:57:51.679567 6925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f"
I20250110 20:57:51.679817 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51"
I20250110 20:57:51.680076 6925 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 3 FOLLOWER]: Advancing to term 5
I20250110 20:57:51.680308 6851 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 4 FOLLOWER]: Advancing to term 5
I20250110 20:57:51.680823 6902 leader_election.cc:400] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 4 pre-election: Vote denied by peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9bb61bdf30044a1b482d6a66e29435f for earlier term 4. Current term is 5.
I20250110 20:57:51.681082 6902 leader_election.cc:403] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20250110 20:57:51.685721 6925 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 5.
I20250110 20:57:51.686059 7426 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621) with higher term. Message: Invalid argument: T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9bb61bdf30044a1b482d6a66e29435f for earlier term 4. Current term is 5.
I20250110 20:57:51.686882 6980 leader_election.cc:304] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f; no voters:
I20250110 20:57:51.687175 6851 raft_consensus.cc:2463] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a36f292a1a20461e9aef5aefe8fd0f08 in term 5.
I20250110 20:57:51.688669 7424 raft_consensus.cc:2798] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 5 FOLLOWER]: Leader election won for term 5
I20250110 20:57:51.689096 7424 raft_consensus.cc:695] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [term 5 LEADER]: Becoming Leader. State: Replica: a36f292a1a20461e9aef5aefe8fd0f08, State: Running, Role: LEADER
I20250110 20:57:51.689916 7424 consensus_queue.cc:237] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:57:51.693393 7423 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:51.694114 7423 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:51.695195 7428 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:57:51.703121 7428 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:57:51.707170 7428 catalog_manager.cc:1260] Loaded cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:57:51.707424 7428 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:57:51.712121 7428 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:57:51.716161 7428 catalog_manager.cc:5910] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Loaded TSK: 0
I20250110 20:57:51.717047 7428 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:57:51.719951 7428 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250110 20:57:51.722579 7428 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: -1
W20250110 20:57:50.736594 7415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250110 20:57:50.738232 7418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:50.737058 7408 server_base.cc:1034] running on GCE node
I20250110 20:57:51.853248 7408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250110 20:57:51.855504 7408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:57:51.856850 7408 hybrid_clock.cc:648] HybridClock initialized: now 1736542671856805 us; error 49 us; skew 500 ppm
I20250110 20:57:51.857862 7408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:57:51.863716 7408 webserver.cc:458] Webserver started at http://127.4.171.195:35785/ using document root <none> and password file <none>
I20250110 20:57:51.864657 7408 fs_manager.cc:362] Metadata directory not provided
I20250110 20:57:51.864842 7408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:57:51.865244 7408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20250110 20:57:51.870302 7408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data/instance:
uuid: "c3b63d95ca734d8a9803b96bfd94f307"
format_stamp: "Formatted at 2025-01-10 20:57:51 on dist-test-slave-98kh"
I20250110 20:57:51.871337 7408 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/instance:
uuid: "c3b63d95ca734d8a9803b96bfd94f307"
format_stamp: "Formatted at 2025-01-10 20:57:51 on dist-test-slave-98kh"
I20250110 20:57:51.878145 7408 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250110 20:57:51.883425 7433 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:51.884393 7408 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20250110 20:57:51.884718 7408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal
uuid: "c3b63d95ca734d8a9803b96bfd94f307"
format_stamp: "Formatted at 2025-01-10 20:57:51 on dist-test-slave-98kh"
I20250110 20:57:51.885015 7408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:57:51.950008 7408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:57:51.951927 7408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:57:51.952461 7408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:57:51.955525 7408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250110 20:57:51.960758 7408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250110 20:57:51.961027 7408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250110 20:57:51.961301 7408 ts_tablet_manager.cc:610] Registered 0 tablets
I20250110 20:57:51.961524 7408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250110 20:57:52.091634 7408 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.195:45717
I20250110 20:57:52.091746 7545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.195:45717 every 8 connection(s)
I20250110 20:57:52.094341 7408 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/data/info.pb
I20250110 20:57:52.101554 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 7408
I20250110 20:57:52.102051 4783 external_mini_cluster.cc:1427] Reading /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/instance
I20250110 20:57:52.121503 7546 heartbeater.cc:346] Connected to a master server at 127.4.171.252:33621
I20250110 20:57:52.121943 7546 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:52.123203 7546 heartbeater.cc:510] Master 127.4.171.252:33621 requested a full tablet report, sending...
I20250110 20:57:52.126029 6994 ts_manager.cc:194] Registered new tserver with Master: c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:52.126202 7548 heartbeater.cc:346] Connected to a master server at 127.4.171.254:36415
I20250110 20:57:52.126514 7548 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:52.127377 7548 heartbeater.cc:510] Master 127.4.171.254:36415 requested a full tablet report, sending...
I20250110 20:57:52.127405 6925 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Refusing update from remote peer a36f292a1a20461e9aef5aefe8fd0f08: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250110 20:57:52.128261 6994 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.171.195:35939
I20250110 20:57:52.128633 7547 heartbeater.cc:346] Connected to a master server at 127.4.171.253:40203
I20250110 20:57:52.128957 7547 heartbeater.cc:463] Registering TS with master...
I20250110 20:57:52.129809 7547 heartbeater.cc:510] Master 127.4.171.253:40203 requested a full tablet report, sending...
I20250110 20:57:52.130273 6842 ts_manager.cc:194] Registered new tserver with Master: c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:52.130125 7423 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 20:57:52.132407 6916 ts_manager.cc:194] Registered new tserver with Master: c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:52.135718 7426 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:52.136473 7426 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:52.143553 6851 raft_consensus.cc:1270] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Refusing update from remote peer a36f292a1a20461e9aef5aefe8fd0f08: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20250110 20:57:52.143371 7424 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:52.144109 7424 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:52.145375 7424 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250110 20:57:52.147302 4783 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250110 20:57:52.150413 7426 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:52.151114 7426 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:52.154937 7562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:52.155241 7559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36f292a1a20461e9aef5aefe8fd0f08. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:52.155786 7559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:52.155735 7562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:57:52.191182 6994 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_9730475044b644c49b5ddc948b0c537d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:52.193244 6992 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_c8e132d9095947f8b4700839d70d8060"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:52.191000 6993 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_52bd7ec724394352a929ba64cab155d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:52.195330 6991 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_a1cbfe0292df4dc8a2c9b8709f0df836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250110 20:57:52.201855 6994 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9730475044b644c49b5ddc948b0c537d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:52.210654 6991 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:52.210745 6993 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_52bd7ec724394352a929ba64cab155d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:52.214152 6992 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_c8e132d9095947f8b4700839d70d8060 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:52.253690 4783 external_mini_cluster.cc:1620] Killing /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu with pid 6878
I20250110 20:57:52.351591 7398 heartbeater.cc:502] Master 127.4.171.252:33621 was elected leader, sending a full tablet report...
W20250110 20:57:52.355746 7399 heartbeater.cc:643] Failed to heartbeat to 127.4.171.253:40203 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
W20250110 20:57:52.629791 6980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:52.631938 7546 heartbeater.cc:502] Master 127.4.171.252:33621 was elected leader, sending a full tablet report...
W20250110 20:57:52.638556 7547 heartbeater.cc:643] Failed to heartbeat to 127.4.171.253:40203 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:52.645991 7255 heartbeater.cc:502] Master 127.4.171.252:33621 was elected leader, sending a full tablet report...
W20250110 20:57:52.645635 6980 consensus_peers.cc:487] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 -> Peer f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203): Couldn't send request to peer f9bb61bdf30044a1b482d6a66e29435f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:52.646660 7423 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:52.648394 7423 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
W20250110 20:57:52.655875 7256 heartbeater.cc:643] Failed to heartbeat to 127.4.171.253:40203 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:57:52.671332 6994 catalog_manager.cc:2149] created HMS catalog entry for table [id=2b85c6667e194f818c41937e2307c37f]
I20250110 20:57:52.707623 6991 catalog_manager.cc:2149] created HMS catalog entry for table [id=6ccc906dcedf49caad8b9efe99546401]
I20250110 20:57:52.716907 6994 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_9730475044b644c49b5ddc948b0c537d" } new_table_name: "default.Table_7a871347a918413297f7da917134a92e" modify_external_catalogs: true
I20250110 20:57:52.730190 6991 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_a1cbfe0292df4dc8a2c9b8709f0df836" } new_table_name: "default.Table_59954cb851e943ff9f10bca64b86d8b8" modify_external_catalogs: true
I20250110 20:57:52.765228 6992 catalog_manager.cc:2149] created HMS catalog entry for table [id=d3ea810c260f466cab0c3f5edb25366d]
I20250110 20:57:52.783324 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 72de597ddab241fbbb82fc04e40385c5 (DEFAULT_TABLE table=default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.785290 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72de597ddab241fbbb82fc04e40385c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.788969 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 69f0a5ceae3a489b9b0552c550d3e80a (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.790148 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0a5ceae3a489b9b0552c550d3e80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.789593 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 72de597ddab241fbbb82fc04e40385c5 (DEFAULT_TABLE table=default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.791273 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72de597ddab241fbbb82fc04e40385c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.803797 6992 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_c8e132d9095947f8b4700839d70d8060" } modify_external_catalogs: true
I20250110 20:57:52.804553 6992 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_c8e132d9095947f8b4700839d70d8060" } modify_external_catalogs: true
I20250110 20:57:52.810217 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 69f0a5ceae3a489b9b0552c550d3e80a (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.811375 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0a5ceae3a489b9b0552c550d3e80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.826714 7602 tablet_bootstrap.cc:492] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:52.835388 7602 tablet_bootstrap.cc:654] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:52.837028 7480 tablet_service.cc:1467] Processing CreateTablet for tablet 69f0a5ceae3a489b9b0552c550d3e80a (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.838904 7480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69f0a5ceae3a489b9b0552c550d3e80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.836902 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 72de597ddab241fbbb82fc04e40385c5 (DEFAULT_TABLE table=default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.840256 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72de597ddab241fbbb82fc04e40385c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.841089 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:52.845388 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:52.839282 7602 log.cc:826] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8: Log is configured to *not* fsync() on all Append() calls
I20250110 20:57:52.867861 7190 tablet_service.cc:1467] Processing CreateTablet for tablet b6f6922d96c34c54b777160cd8d5c0d0 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.868942 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6f6922d96c34c54b777160cd8d5c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.873235 6993 catalog_manager.cc:2149] created HMS catalog entry for table [id=6526e5cacb274d1097e6d2e6e4f973c2]
I20250110 20:57:52.873971 7606 tablet_bootstrap.cc:492] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:52.881991 4783 external_mini_cluster.cc:1351] Running /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
/tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.253:40203
--webserver_interface=127.4.171.253
--webserver_port=33297
--builtin_ntp_servers=127.4.171.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--hive_metastore_uris=thrift://127.0.0.1:43427
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250110 20:57:52.892608 7606 tablet_bootstrap.cc:654] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:52.895113 7606 log.cc:826] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Log is configured to *not* fsync() on all Append() calls
I20250110 20:57:52.911777 7606 tablet_bootstrap.cc:492] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:52.912391 7606 ts_tablet_manager.cc:1397] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.004s
I20250110 20:57:52.934435 7609 tablet_bootstrap.cc:492] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:52.963995 7602 tablet_bootstrap.cc:492] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:52.964620 7602 ts_tablet_manager.cc:1397] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.138s user 0.040s sys 0.064s
I20250110 20:57:52.972512 7606 raft_consensus.cc:357] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:52.973623 7606 raft_consensus.cc:383] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:52.973978 7606 raft_consensus.cc:738] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:52.975075 7480 tablet_service.cc:1467] Processing CreateTablet for tablet b6f6922d96c34c54b777160cd8d5c0d0 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:52.976146 7480 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6f6922d96c34c54b777160cd8d5c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:52.981076 7606 consensus_queue.cc:260] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:52.994143 7606 ts_tablet_manager.cc:1428] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.081s user 0.022s sys 0.012s
I20250110 20:57:52.995252 7606 tablet_bootstrap.cc:492] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:52.994318 6987 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_045d0574736e4b9e8fdf9e6a91066d06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:52.996528 6986 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_0433916f941242f994c050f80847e23e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:52.998518 7602 raft_consensus.cc:357] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:52.999585 7602 raft_consensus.cc:383] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:52.999950 7602 raft_consensus.cc:738] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.000856 7602 consensus_queue.cc:260] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.005270 7606 tablet_bootstrap.cc:654] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.005568 6988 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_391e103e657c4fa2914f89bf58d900be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:53.016362 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.018204 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 72de597ddab241fbbb82fc04e40385c5 deleted and replaced by tablet 23b9983db2c649ea964e57a4dff784de
I20250110 20:57:53.019877 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.021162 6989 catalog_manager.cc:6596] ReplaceTablet: tablet 69f0a5ceae3a489b9b0552c550d3e80a deleted and replaced by tablet 167292984bbc45faaabd68f0aee8212b
I20250110 20:57:53.021085 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 72de597ddab241fbbb82fc04e40385c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 23b9983db2c649ea964e57a4dff784de) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:53.026741 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f0a5ceae3a489b9b0552c550d3e80a with delete_type TABLET_DATA_DELETED (replaced by tablet 167292984bbc45faaabd68f0aee8212b) from {username='slave'} at 127.0.0.1:42840
W20250110 20:57:53.012784 7288 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.Ping -> {remote=127.4.171.253:40203, user_credentials={real_user=slave}} blocked reactor thread for 51229.7us
I20250110 20:57:53.039943 7333 tablet_service.cc:1467] Processing CreateTablet for tablet b6f6922d96c34c54b777160cd8d5c0d0 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.040956 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6f6922d96c34c54b777160cd8d5c0d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.048059 7610 raft_consensus.cc:491] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:53.048581 7610 raft_consensus.cc:513] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.053583 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f0a5ceae3a489b9b0552c550d3e80a with delete_type TABLET_DATA_DELETED (replaced by tablet 167292984bbc45faaabd68f0aee8212b) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:53.057507 7610 leader_election.cc:290] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:57:53.057560 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet 72de597ddab241fbbb82fc04e40385c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72de597ddab241fbbb82fc04e40385c5 already in progress: creating tablet
W20250110 20:57:53.058513 7401 tablet.cc:2367] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250110 20:57:53.061964 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet 69f0a5ceae3a489b9b0552c550d3e80a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f0a5ceae3a489b9b0552c550d3e80a already in progress: creating tablet
I20250110 20:57:53.064874 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72de597ddab241fbbb82fc04e40385c5" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:53.066148 7479 tablet_service.cc:1514] Processing DeleteTablet for tablet 72de597ddab241fbbb82fc04e40385c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 23b9983db2c649ea964e57a4dff784de) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:53.066805 7606 tablet_bootstrap.cc:492] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
W20250110 20:57:53.066895 7287 leader_election.cc:343] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:53.067301 7606 ts_tablet_manager.cc:1397] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.004s
I20250110 20:57:53.069959 7606 raft_consensus.cc:357] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.070700 7606 raft_consensus.cc:383] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.071009 7606 raft_consensus.cc:738] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.071753 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 72de597ddab241fbbb82fc04e40385c5 with delete_type TABLET_DATA_DELETED (replaced by tablet 23b9983db2c649ea964e57a4dff784de) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:53.071638 7606 consensus_queue.cc:260] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.073994 7606 ts_tablet_manager.cc:1428] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250110 20:57:53.074801 7606 tablet_bootstrap.cc:492] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:53.080672 7606 tablet_bootstrap.cc:654] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.085347 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 69f0a5ceae3a489b9b0552c550d3e80a with delete_type TABLET_DATA_DELETED (replaced by tablet 167292984bbc45faaabd68f0aee8212b) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:53.095520 6993 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_52bd7ec724394352a929ba64cab155d2" } modify_external_catalogs: true
I20250110 20:57:53.096385 6993 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_52bd7ec724394352a929ba64cab155d2" } modify_external_catalogs: true
I20250110 20:57:53.102484 7597 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:57:53.103437 7597 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 [sys.catalog]: This master's current role is: LEADER
I20250110 20:57:53.151705 7617 tablet_replica.cc:331] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:53.152690 7617 raft_consensus.cc:2238] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:53.153357 7617 raft_consensus.cc:2267] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:53.167526 7189 tablet_service.cc:1467] Processing CreateTablet for tablet ee0cb74505814d279c741ced75a30e93 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:53.168015 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.168641 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0cb74505814d279c741ced75a30e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.171547 7479 tablet_service.cc:1467] Processing CreateTablet for tablet ee0cb74505814d279c741ced75a30e93 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.173729 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 167292984bbc45faaabd68f0aee8212b (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.174712 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167292984bbc45faaabd68f0aee8212b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.175050 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 167292984bbc45faaabd68f0aee8212b (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:57:53.175453 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.178551 7609 tablet_bootstrap.cc:654] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.180757 7609 log.cc:826] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307: Log is configured to *not* fsync() on all Append() calls
I20250110 20:57:53.181835 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 23b9983db2c649ea964e57a4dff784de (DEFAULT_TABLE table=default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.182965 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b9983db2c649ea964e57a4dff784de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.182785 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 23b9983db2c649ea964e57a4dff784de (DEFAULT_TABLE table=default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.184619 7617 ts_tablet_manager.cc:1905] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:53.187201 7479 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0cb74505814d279c741ced75a30e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.188731 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 167292984bbc45faaabd68f0aee8212b (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.189157 7602 ts_tablet_manager.cc:1428] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.224s user 0.050s sys 0.036s
I20250110 20:57:53.189095 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 23b9983db2c649ea964e57a4dff784de (DEFAULT_TABLE table=default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.189919 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167292984bbc45faaabd68f0aee8212b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.192708 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b9983db2c649ea964e57a4dff784de. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:53.197211 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.203765 7602 tablet_bootstrap.cc:492] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:53.181304 7332 tablet_service.cc:1467] Processing CreateTablet for tablet ee0cb74505814d279c741ced75a30e93 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.210085 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee0cb74505814d279c741ced75a30e93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.214365 7617 ts_tablet_manager.cc:1918] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:53.215102 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167292984bbc45faaabd68f0aee8212b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.220770 7602 tablet_bootstrap.cc:654] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.232677 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.241075 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.258514 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b9983db2c649ea964e57a4dff784de. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.258695 7606 tablet_bootstrap.cc:492] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:53.261739 7606 ts_tablet_manager.cc:1397] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.187s user 0.019s sys 0.020s
I20250110 20:57:53.264322 7606 raft_consensus.cc:357] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.265024 7606 raft_consensus.cc:383] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.265358 7606 raft_consensus.cc:738] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.266024 7606 consensus_queue.cc:260] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.269081 7617 log.cc:1198] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.270051 7617 ts_tablet_manager.cc:1939] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:53.271950 7617 tablet_replica.cc:331] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:53.272127 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 72de597ddab241fbbb82fc04e40385c5 (table default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:53.272539 7617 raft_consensus.cc:2238] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:53.272994 7617 raft_consensus.cc:2267] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:53.285027 7606 ts_tablet_manager.cc:1428] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250110 20:57:53.286026 7606 tablet_bootstrap.cc:492] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:53.291750 7606 tablet_bootstrap.cc:654] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:53.294492 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 69f0a5ceae3a489b9b0552c550d3e80a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69f0a5ceae3a489b9b0552c550d3e80a already in progress: creating tablet
W20250110 20:57:53.296068 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 72de597ddab241fbbb82fc04e40385c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72de597ddab241fbbb82fc04e40385c5 already in progress: creating tablet
I20250110 20:57:53.277379 7609 tablet_bootstrap.cc:492] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:53.298228 7609 ts_tablet_manager.cc:1397] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.364s user 0.010s sys 0.009s
I20250110 20:57:53.300087 6985 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_df681f6d6d9e483282a9b28e42817aef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:53.311017 7602 tablet_bootstrap.cc:492] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:53.311429 7602 ts_tablet_manager.cc:1397] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.007s
I20250110 20:57:53.314144 7602 raft_consensus.cc:357] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.314915 7602 raft_consensus.cc:383] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.315230 7602 raft_consensus.cc:738] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.315928 7602 consensus_queue.cc:260] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.319679 7602 ts_tablet_manager.cc:1428] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.008s user 0.004s sys 0.001s
I20250110 20:57:53.320564 7602 tablet_bootstrap.cc:492] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
W20250110 20:57:53.323303 7262 tablet.cc:2367] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:57:53.311059 7617 ts_tablet_manager.cc:1905] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:53.330910 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:57:53.335705 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.317638 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72de597ddab241fbbb82fc04e40385c5" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:53.349786 7602 tablet_bootstrap.cc:654] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:53.350090 7288 leader_election.cc:343] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:53.350522 7288 leader_election.cc:304] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
W20250110 20:57:53.351238 7288 raft_consensus.cc:2681] T 72de597ddab241fbbb82fc04e40385c5 P 9ef4ce3d44cc486fb0ee0ad85127011b: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250110 20:57:53.371296 7606 tablet_bootstrap.cc:492] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:53.371887 7606 ts_tablet_manager.cc:1397] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.003s
I20250110 20:57:53.374493 7606 raft_consensus.cc:357] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.381551 7606 raft_consensus.cc:383] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.381938 7606 raft_consensus.cc:738] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.382668 7606 consensus_queue.cc:260] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.385805 7609 raft_consensus.cc:357] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.386736 7609 raft_consensus.cc:383] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.387063 7609 raft_consensus.cc:738] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.387981 7609 consensus_queue.cc:260] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.389657 7602 tablet_bootstrap.cc:492] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:53.390123 7602 ts_tablet_manager.cc:1397] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.005s
I20250110 20:57:53.392590 7602 raft_consensus.cc:357] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.398478 7606 ts_tablet_manager.cc:1428] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250110 20:57:53.404443 7602 raft_consensus.cc:383] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.404839 7602 raft_consensus.cc:738] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.380079 7617 ts_tablet_manager.cc:1918] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:53.405570 7617 log.cc:1198] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.406324 7617 ts_tablet_manager.cc:1939] T 69f0a5ceae3a489b9b0552c550d3e80a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:53.409109 7602 consensus_queue.cc:260] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.410693 7609 ts_tablet_manager.cc:1428] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.112s user 0.026s sys 0.014s
I20250110 20:57:53.408646 7606 tablet_bootstrap.cc:492] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:53.411588 7609 tablet_bootstrap.cc:492] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:53.418644 7602 ts_tablet_manager.cc:1428] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250110 20:57:53.422147 7602 tablet_bootstrap.cc:492] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:53.423301 7609 tablet_bootstrap.cc:654] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.428826 7611 raft_consensus.cc:491] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:53.429344 7611 raft_consensus.cc:513] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.432353 7611 leader_election.cc:290] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:53.438481 7606 tablet_bootstrap.cc:654] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:53.450562 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.455951 7602 tablet_bootstrap.cc:654] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.456763 7609 tablet_bootstrap.cc:492] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:53.457183 7609 ts_tablet_manager.cc:1397] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.000s
I20250110 20:57:53.457563 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet ee0cb74505814d279c741ced75a30e93 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.459731 7609 raft_consensus.cc:357] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.460402 7609 raft_consensus.cc:383] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.460811 7609 raft_consensus.cc:738] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.461489 7609 consensus_queue.cc:260] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.463327 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 69f0a5ceae3a489b9b0552c550d3e80a (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:53.464520 7609 ts_tablet_manager.cc:1428] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:57:53.468622 7609 tablet_bootstrap.cc:492] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
W20250110 20:57:53.470752 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.480965 7609 tablet_bootstrap.cc:654] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.483425 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet ee0cb74505814d279c741ced75a30e93
I20250110 20:57:53.485500 7606 tablet_bootstrap.cc:492] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:53.488772 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0cb74505814d279c741ced75a30e93 with delete_type TABLET_DATA_DELETED (replaced by tablet 7128e3c4f59141b0943f03ba7261ff04) from {username='slave'} at 127.0.0.1:40394
W20250110 20:57:53.490720 6980 catalog_manager.cc:4852] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet ee0cb74505814d279c741ced75a30e93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee0cb74505814d279c741ced75a30e93 already in progress: creating tablet
I20250110 20:57:53.491601 7606 ts_tablet_manager.cc:1397] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.004s
I20250110 20:57:53.488808 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0cb74505814d279c741ced75a30e93 with delete_type TABLET_DATA_DELETED (replaced by tablet 7128e3c4f59141b0943f03ba7261ff04) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:53.494201 7606 raft_consensus.cc:357] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.497172 6990 catalog_manager.cc:6596] ReplaceTablet: tablet ee0cb74505814d279c741ced75a30e93 deleted and replaced by tablet 7128e3c4f59141b0943f03ba7261ff04
W20250110 20:57:53.495831 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet ee0cb74505814d279c741ced75a30e93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee0cb74505814d279c741ced75a30e93 already in progress: creating tablet
I20250110 20:57:53.501731 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.506486 7602 tablet_bootstrap.cc:492] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:53.509860 7606 raft_consensus.cc:383] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.517994 7609 tablet_bootstrap.cc:492] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:53.518462 7609 ts_tablet_manager.cc:1397] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.002s
I20250110 20:57:53.521025 7609 raft_consensus.cc:357] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.521875 7609 raft_consensus.cc:383] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.523237 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72de597ddab241fbbb82fc04e40385c5" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:53.522182 7609 raft_consensus.cc:738] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.524395 7609 consensus_queue.cc:260] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.525226 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 7128e3c4f59141b0943f03ba7261ff04 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.526253 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7128e3c4f59141b0943f03ba7261ff04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.522641 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 7128e3c4f59141b0943f03ba7261ff04 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.527437 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7128e3c4f59141b0943f03ba7261ff04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.529582 7609 ts_tablet_manager.cc:1428] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250110 20:57:53.530484 7609 tablet_bootstrap.cc:492] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:53.521064 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0cb74505814d279c741ced75a30e93 with delete_type TABLET_DATA_DELETED (replaced by tablet 7128e3c4f59141b0943f03ba7261ff04) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:53.525497 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 7128e3c4f59141b0943f03ba7261ff04 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:53.538342 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7128e3c4f59141b0943f03ba7261ff04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:53.541105 7609 tablet_bootstrap.cc:654] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:53.526418 7145 leader_election.cc:343] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5
W20250110 20:57:53.545397 7145 leader_election.cc:343] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:53.545846 7606 raft_consensus.cc:738] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.546739 7145 leader_election.cc:304] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:53.547636 7606 consensus_queue.cc:260] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.546442 7602 ts_tablet_manager.cc:1397] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.124s user 0.009s sys 0.003s
I20250110 20:57:53.554692 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0cb74505814d279c741ced75a30e93 with delete_type TABLET_DATA_DELETED (replaced by tablet 7128e3c4f59141b0943f03ba7261ff04) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:53.555537 7616 tablet_replica.cc:331] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:53.556332 7616 raft_consensus.cc:2238] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:53.556854 7616 raft_consensus.cc:2267] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:53.559211 7616 ts_tablet_manager.cc:1905] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:53.561338 7609 tablet_bootstrap.cc:492] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:53.561812 7609 ts_tablet_manager.cc:1397] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.004s
I20250110 20:57:53.564277 7609 raft_consensus.cc:357] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.565085 7609 raft_consensus.cc:383] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.565379 7609 raft_consensus.cc:738] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.566115 7609 consensus_queue.cc:260] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.569746 7611 raft_consensus.cc:2743] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:53.571130 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet ee0cb74505814d279c741ced75a30e93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee0cb74505814d279c741ced75a30e93 already in progress: creating tablet
W20250110 20:57:53.574678 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.574520 7609 ts_tablet_manager.cc:1428] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.012s user 0.001s sys 0.004s
I20250110 20:57:53.576489 7609 tablet_bootstrap.cc:492] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:53.578936 7606 ts_tablet_manager.cc:1428] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.087s user 0.005s sys 0.000s
I20250110 20:57:53.579803 7606 tablet_bootstrap.cc:492] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:53.586230 7602 raft_consensus.cc:357] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.593607 7626 raft_consensus.cc:491] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:53.594200 7626 raft_consensus.cc:513] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.595676 7606 tablet_bootstrap.cc:654] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.586933 7602 raft_consensus.cc:383] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.598562 7602 raft_consensus.cc:738] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.599196 7602 consensus_queue.cc:260] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.601063 7626 leader_election.cc:290] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:57:53.604015 7549 tablet.cc:2367] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250110 20:57:53.604362 7609 tablet_bootstrap.cc:654] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:53.604975 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.610208 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0cb74505814d279c741ced75a30e93 with delete_type TABLET_DATA_DELETED (replaced by tablet 7128e3c4f59141b0943f03ba7261ff04) from {username='slave'} at 127.0.0.1:42840
W20250110 20:57:53.611636 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet ee0cb74505814d279c741ced75a30e93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ee0cb74505814d279c741ced75a30e93 already in progress: creating tablet
I20250110 20:57:53.612196 7610 raft_consensus.cc:491] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:53.612608 7610 raft_consensus.cc:513] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.614429 7610 leader_election.cc:290] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:53.615799 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b9983db2c649ea964e57a4dff784de" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
W20250110 20:57:53.617499 7287 leader_election.cc:343] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:53.618544 7609 tablet_bootstrap.cc:492] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:53.619848 7609 ts_tablet_manager.cc:1397] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.044s user 0.006s sys 0.006s
I20250110 20:57:53.625834 7602 ts_tablet_manager.cc:1428] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.073s user 0.006s sys 0.000s
I20250110 20:57:53.628857 7609 raft_consensus.cc:357] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.630302 7609 raft_consensus.cc:383] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.630942 7609 raft_consensus.cc:738] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.633313 7609 consensus_queue.cc:260] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.635530 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b9983db2c649ea964e57a4dff784de" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:53.638474 7501 raft_consensus.cc:2463] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:53.645310 7609 ts_tablet_manager.cc:1428] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.025s user 0.002s sys 0.003s
I20250110 20:57:53.645970 7602 tablet_bootstrap.cc:492] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:53.645900 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0cb74505814d279c741ced75a30e93 with delete_type TABLET_DATA_DELETED (replaced by tablet 7128e3c4f59141b0943f03ba7261ff04) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:53.647691 7615 tablet_replica.cc:331] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:53.647894 7609 tablet_bootstrap.cc:492] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:53.648479 7615 raft_consensus.cc:2238] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:53.648958 7615 raft_consensus.cc:2267] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:53.651958 7615 ts_tablet_manager.cc:1905] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:53.658947 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72de597ddab241fbbb82fc04e40385c5" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:53.659554 7210 raft_consensus.cc:2463] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:53.661026 7609 tablet_bootstrap.cc:654] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.663218 7606 tablet_bootstrap.cc:492] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:53.663709 7606 ts_tablet_manager.cc:1397] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.001s
I20250110 20:57:53.660892 7435 leader_election.cc:304] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:53.666321 7606 raft_consensus.cc:357] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.666772 7626 raft_consensus.cc:2798] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:53.666716 7288 leader_election.cc:304] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:57:53.667114 7626 raft_consensus.cc:491] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:53.667408 7626 raft_consensus.cc:3054] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.667487 7610 raft_consensus.cc:2798] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:53.667807 7610 raft_consensus.cc:491] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:53.668148 7610 raft_consensus.cc:3054] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.670787 7616 ts_tablet_manager.cc:1918] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:53.671172 7616 log.cc:1198] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ee0cb74505814d279c741ced75a30e93
I20250110 20:57:53.672005 7616 ts_tablet_manager.cc:1939] T ee0cb74505814d279c741ced75a30e93 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
W20250110 20:57:53.674268 7436 leader_election.cc:343] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.676029 7626 raft_consensus.cc:513] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
W20250110 20:57:53.678728 7436 leader_election.cc:343] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.679314 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72de597ddab241fbbb82fc04e40385c5" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:53.679214 7610 raft_consensus.cc:513] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.679733 7210 raft_consensus.cc:3054] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.680990 7610 leader_election.cc:290] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:53.667105 7606 raft_consensus.cc:383] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.681712 7606 raft_consensus.cc:738] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.682229 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b9983db2c649ea964e57a4dff784de" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:53.682346 7606 consensus_queue.cc:260] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
W20250110 20:57:53.683702 7287 leader_election.cc:343] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:53.685314 7210 raft_consensus.cc:2463] T 72de597ddab241fbbb82fc04e40385c5 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:53.686261 7435 leader_election.cc:304] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:57:53.688727 7602 tablet_bootstrap.cc:654] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.690133 7626 leader_election.cc:290] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:57:53.690528 7606 ts_tablet_manager.cc:1428] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250110 20:57:53.691040 7626 raft_consensus.cc:2798] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:53.693514 7626 raft_consensus.cc:695] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:53.697052 7606 tablet_bootstrap.cc:492] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:53.698050 7626 consensus_queue.cc:237] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.705770 7606 tablet_bootstrap.cc:654] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.708942 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b9983db2c649ea964e57a4dff784de" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:53.709515 7501 raft_consensus.cc:3054] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.710103 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet ee0cb74505814d279c741ced75a30e93 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:53.714195 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.722007 7615 ts_tablet_manager.cc:1918] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:53.722520 7615 log.cc:1198] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ee0cb74505814d279c741ced75a30e93
I20250110 20:57:53.723385 7615 ts_tablet_manager.cc:1939] T ee0cb74505814d279c741ced75a30e93 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:53.724766 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet ee0cb74505814d279c741ced75a30e93 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:53.732185 7609 tablet_bootstrap.cc:492] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:53.732690 7609 ts_tablet_manager.cc:1397] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.007s
I20250110 20:57:53.735634 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet ee0cb74505814d279c741ced75a30e93 with delete_type TABLET_DATA_DELETED (replaced by tablet 7128e3c4f59141b0943f03ba7261ff04) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:53.735715 7609 raft_consensus.cc:357] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.736356 7617 tablet_replica.cc:331] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:53.736382 7609 raft_consensus.cc:383] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.736630 7609 raft_consensus.cc:738] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.737064 7617 raft_consensus.cc:2238] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:53.737510 7617 raft_consensus.cc:2267] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:53.739845 7617 ts_tablet_manager.cc:1905] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:53.748961 7602 tablet_bootstrap.cc:492] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:53.749480 7602 ts_tablet_manager.cc:1397] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.002s
I20250110 20:57:53.749926 7501 raft_consensus.cc:2463] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:53.751716 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.753032 7602 raft_consensus.cc:357] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.753731 7602 raft_consensus.cc:383] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.753775 7288 leader_election.cc:304] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:57:53.754002 7602 raft_consensus.cc:738] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.754611 7610 raft_consensus.cc:2798] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:53.754606 7602 consensus_queue.cc:260] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.737238 7609 consensus_queue.cc:260] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.766499 7611 raft_consensus.cc:491] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:53.766897 7611 raft_consensus.cc:513] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.768632 7611 leader_election.cc:290] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:53.770390 7617 ts_tablet_manager.cc:1918] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:53.770889 7617 log.cc:1198] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ee0cb74505814d279c741ced75a30e93
I20250110 20:57:53.771588 7617 ts_tablet_manager.cc:1939] T ee0cb74505814d279c741ced75a30e93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:53.772619 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f6922d96c34c54b777160cd8d5c0d0" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:53.773345 7353 raft_consensus.cc:2463] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:53.774663 7145 leader_election.cc:304] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:53.775800 7611 raft_consensus.cc:2798] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:53.776146 7611 raft_consensus.cc:491] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:53.776420 7611 raft_consensus.cc:3054] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.776683 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet ee0cb74505814d279c741ced75a30e93 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:53.777136 7609 ts_tablet_manager.cc:1428] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20250110 20:57:53.778035 7609 tablet_bootstrap.cc:492] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:53.781838 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f6922d96c34c54b777160cd8d5c0d0" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:53.783468 7611 raft_consensus.cc:513] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.784339 7501 raft_consensus.cc:2463] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:53.785907 7606 tablet_bootstrap.cc:492] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:53.786295 7606 ts_tablet_manager.cc:1397] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.095s user 0.009s sys 0.008s
I20250110 20:57:53.787582 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f6922d96c34c54b777160cd8d5c0d0" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:53.788086 7353 raft_consensus.cc:3054] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.788468 7606 raft_consensus.cc:357] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.789125 7606 raft_consensus.cc:383] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.789357 7606 raft_consensus.cc:738] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.790205 7602 ts_tablet_manager.cc:1428] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20250110 20:57:53.791128 7602 tablet_bootstrap.cc:492] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:53.790352 7606 consensus_queue.cc:260] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.801571 7353 raft_consensus.cc:2463] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:53.802561 7145 leader_election.cc:304] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:53.803633 7611 leader_election.cc:290] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:53.804194 7611 raft_consensus.cc:2798] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:53.805158 7646 raft_consensus.cc:491] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:53.805897 7611 raft_consensus.cc:695] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:57:53.807438 7611 consensus_queue.cc:237] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.808537 7606 ts_tablet_manager.cc:1428] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20250110 20:57:53.806649 7646 raft_consensus.cc:513] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.813035 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167292984bbc45faaabd68f0aee8212b" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
W20250110 20:57:53.814895 7287 leader_election.cc:343] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:53.818142 7609 tablet_bootstrap.cc:654] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.818830 7646 leader_election.cc:290] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:53.821959 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167292984bbc45faaabd68f0aee8212b" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:53.822629 7501 raft_consensus.cc:2463] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:53.825899 7288 leader_election.cc:304] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:57:53.827401 7646 raft_consensus.cc:2798] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:53.827860 7646 raft_consensus.cc:491] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:53.828213 7602 tablet_bootstrap.cc:654] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.828212 7646 raft_consensus.cc:3054] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.830575 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6f6922d96c34c54b777160cd8d5c0d0" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:53.830823 7610 raft_consensus.cc:695] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:53.831220 7501 raft_consensus.cc:3054] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.831701 7610 consensus_queue.cc:237] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.846943 6990 catalog_manager.cc:5526] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
W20250110 20:57:53.849983 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:53.837322 7501 raft_consensus.cc:2463] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:53.852689 7609 tablet_bootstrap.cc:492] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:53.853078 7609 ts_tablet_manager.cc:1397] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.007s
I20250110 20:57:53.855291 7609 raft_consensus.cc:357] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.857190 7609 raft_consensus.cc:383] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.858935 7602 tablet_bootstrap.cc:492] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:53.859390 7602 ts_tablet_manager.cc:1397] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.068s user 0.010s sys 0.004s
I20250110 20:57:53.861764 7609 raft_consensus.cc:738] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.854532 7646 raft_consensus.cc:513] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.862582 7602 raft_consensus.cc:357] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.863636 7609 consensus_queue.cc:260] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.864089 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167292984bbc45faaabd68f0aee8212b" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:53.864800 7602 raft_consensus.cc:383] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.864843 7646 leader_election.cc:290] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:53.866024 7602 raft_consensus.cc:738] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.868074 7609 ts_tablet_manager.cc:1428] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.015s user 0.002s sys 0.005s
I20250110 20:57:53.867542 7602 consensus_queue.cc:260] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.872313 7210 raft_consensus.cc:3054] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.877938 7210 raft_consensus.cc:2463] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:53.884366 7602 ts_tablet_manager.cc:1428] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20250110 20:57:53.885269 7602 tablet_bootstrap.cc:492] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:53.890771 7602 tablet_bootstrap.cc:654] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:53.898110 7626 raft_consensus.cc:491] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:53.898558 7626 raft_consensus.cc:513] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.900416 7626 leader_election.cc:290] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:53.902099 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f0a5ceae3a489b9b0552c550d3e80a" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:53.902670 7210 raft_consensus.cc:2463] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:53.904058 7435 leader_election.cc:304] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:53.904876 7626 raft_consensus.cc:2798] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:53.905238 7626 raft_consensus.cc:491] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:53.905652 7626 raft_consensus.cc:3054] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.909927 7287 leader_election.cc:304] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:53.911682 7646 raft_consensus.cc:2798] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:53.912137 7646 raft_consensus.cc:695] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:53.912894 7646 consensus_queue.cc:237] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:53.923898 7626 raft_consensus.cc:513] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.927178 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.928962 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "69f0a5ceae3a489b9b0552c550d3e80a" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:53.929525 7210 raft_consensus.cc:3054] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:53.930543 7436 leader_election.cc:343] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.930611 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167292984bbc45faaabd68f0aee8212b" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:53.931146 7501 raft_consensus.cc:3054] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:53.935520 7210 raft_consensus.cc:2463] T 69f0a5ceae3a489b9b0552c550d3e80a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:53.936650 7435 leader_election.cc:304] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:53.937712 7639 raft_consensus.cc:2798] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 20:57:53.941536 7436 leader_election.cc:343] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:53.942777 7626 leader_election.cc:290] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:53.944594 7501 raft_consensus.cc:2463] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:53.956843 7602 tablet_bootstrap.cc:492] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:53.957226 7602 ts_tablet_manager.cc:1397] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.072s user 0.012s sys 0.004s
I20250110 20:57:53.957155 7639 raft_consensus.cc:695] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:53.958063 7639 consensus_queue.cc:237] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:53.960336 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:53.960490 7602 raft_consensus.cc:357] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.968350 7602 raft_consensus.cc:383] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:53.968684 7602 raft_consensus.cc:738] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:53.969185 6989 catalog_manager.cc:5526] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:53.969720 7602 consensus_queue.cc:260] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:53.972429 7602 ts_tablet_manager.cc:1428] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20250110 20:57:54.001654 6989 catalog_manager.cc:5526] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } }
W20250110 20:57:54.007129 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:54.042749 7611 raft_consensus.cc:491] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:54.043200 7611 raft_consensus.cc:513] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:57:54.044914 7611 leader_election.cc:290] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:54.046692 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b9983db2c649ea964e57a4dff784de" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:54.047449 7501 raft_consensus.cc:2388] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 244d9ba396f549f2a5015792b6c5fff8 in current term 1: Already voted for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in this term.
I20250110 20:57:54.049006 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23b9983db2c649ea964e57a4dff784de" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:54.050935 7145 leader_election.cc:304] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:54.053344 7611 raft_consensus.cc:3054] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:54.056773 7607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250110 20:57:54.058435 7607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250110 20:57:54.059129 7607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250110 20:57:54.059255 7611 raft_consensus.cc:2743] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:54.105893 6994 catalog_manager.cc:3422] renamed table default.table_9730475044b644c49b5ddc948b0c537d [id=2b85c6667e194f818c41937e2307c37f] in HMS: new name default.table_7a871347a918413297f7da917134a92e
W20250110 20:57:54.119894 7607 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20250110 20:57:54.120241 7607 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20250110 20:57:54.120692 7607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250110 20:57:54.121050 7607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250110 20:57:54.121383 7607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250110 20:57:54.121722 7607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250110 20:57:54.136309 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 7128e3c4f59141b0943f03ba7261ff04 from {username='slave'} at 127.0.0.1:33774
W20250110 20:57:54.138525 6987 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_045d0574736e4b9e8fdf9e6a91066d06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:54.142426 6986 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_0433916f941242f994c050f80847e23e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:54.146945 6988 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_391e103e657c4fa2914f89bf58d900be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:54.169090 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 23b9983db2c649ea964e57a4dff784de from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:54.177170 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 7128e3c4f59141b0943f03ba7261ff04
I20250110 20:57:54.180708 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet 7128e3c4f59141b0943f03ba7261ff04 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c414aabf453406fb42216a6d4121882) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:54.181493 7615 tablet_replica.cc:331] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:54.182281 7615 raft_consensus.cc:2238] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.182721 7615 raft_consensus.cc:2267] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.184480 6989 catalog_manager.cc:6596] ReplaceTablet: tablet 7128e3c4f59141b0943f03ba7261ff04 deleted and replaced by tablet 3c414aabf453406fb42216a6d4121882
I20250110 20:57:54.185863 7615 ts_tablet_manager.cc:1905] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.188287 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 7128e3c4f59141b0943f03ba7261ff04 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c414aabf453406fb42216a6d4121882) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:54.188462 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 7128e3c4f59141b0943f03ba7261ff04 with delete_type TABLET_DATA_DELETED (replaced by tablet 3c414aabf453406fb42216a6d4121882) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:54.188982 7617 tablet_replica.cc:331] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:54.189116 7616 tablet_replica.cc:331] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:54.189697 7617 raft_consensus.cc:2238] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.190059 7616 raft_consensus.cc:2238] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.190151 7617 raft_consensus.cc:2267] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.190491 7616 raft_consensus.cc:2267] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.192234 7617 ts_tablet_manager.cc:1905] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.192698 7616 ts_tablet_manager.cc:1905] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.202805 7501 raft_consensus.cc:1270] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:54.204552 7210 raft_consensus.cc:1270] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:54.206223 7610 consensus_queue.cc:1035] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:54.208107 7615 ts_tablet_manager.cc:1918] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.208464 7615 log.cc:1198] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/7128e3c4f59141b0943f03ba7261ff04
I20250110 20:57:54.208247 7610 consensus_queue.cc:1035] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:54.209129 7615 ts_tablet_manager.cc:1939] T 7128e3c4f59141b0943f03ba7261ff04 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:54.217674 7616 ts_tablet_manager.cc:1918] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.220764 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 7128e3c4f59141b0943f03ba7261ff04 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:54.220916 7616 log.cc:1198] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/7128e3c4f59141b0943f03ba7261ff04
I20250110 20:57:54.221889 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 23b9983db2c649ea964e57a4dff784de
I20250110 20:57:54.223223 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 23b9983db2c649ea964e57a4dff784de deleted and replaced by tablet 5a955851581349debfc895eed2411635
I20250110 20:57:54.222007 7616 ts_tablet_manager.cc:1939] T 7128e3c4f59141b0943f03ba7261ff04 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:54.226503 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b9983db2c649ea964e57a4dff784de with delete_type TABLET_DATA_DELETED (replaced by tablet 5a955851581349debfc895eed2411635) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:54.227217 7615 tablet_replica.cc:331] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:54.229307 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 3c414aabf453406fb42216a6d4121882 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.230369 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c414aabf453406fb42216a6d4121882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.235282 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b9983db2c649ea964e57a4dff784de with delete_type TABLET_DATA_DELETED (replaced by tablet 5a955851581349debfc895eed2411635) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:54.235579 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 23b9983db2c649ea964e57a4dff784de with delete_type TABLET_DATA_DELETED (replaced by tablet 5a955851581349debfc895eed2411635) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:54.236137 7616 tablet_replica.cc:331] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:54.236733 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 3c414aabf453406fb42216a6d4121882 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.237775 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c414aabf453406fb42216a6d4121882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.238324 7607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.171.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:43427
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.171.254:36415,127.4.171.253:40203,127.4.171.252:33621
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.171.253:40203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.171.253
--webserver_port=33297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision bda246888fb83d6b8131ddbb04f062f3e03180a8
build type FASTDEBUG
built by None at 10 Jan 2025 20:43:23 UTC on a633016ff5c9
build id 4116
TSAN enabled
I20250110 20:57:54.238371 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 3c414aabf453406fb42216a6d4121882 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.239375 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c414aabf453406fb42216a6d4121882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.239988 7607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250110 20:57:54.242106 7607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250110 20:57:54.247363 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 7128e3c4f59141b0943f03ba7261ff04 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:54.257306 7609 tablet_bootstrap.cc:492] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:54.262663 7609 tablet_bootstrap.cc:654] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.270507 7606 tablet_bootstrap.cc:492] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:54.272234 7615 raft_consensus.cc:2238] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.274201 7615 pending_rounds.cc:62] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 2 pending ops.
I20250110 20:57:54.274670 7615 pending_rounds.cc:69] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878793476296704 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "23b9983db2c649ea964e57a4dff784de" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_7a871347a918413297f7da917134a92e" dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" new_extra_config { } }
I20250110 20:57:54.276087 7615 pending_rounds.cc:69] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878792378056704 op_type: NO_OP noop_request { }
I20250110 20:57:54.276574 7615 raft_consensus.cc:2883] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:54.277705 7615 raft_consensus.cc:2267] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.279600 7606 tablet_bootstrap.cc:654] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.284174 7662 mvcc.cc:204] Tried to move back new op lower bound from 7112878793476296704 to 7112878792378056704. Current Snapshot: MvccSnapshot[applied={T|T < 7112878793476296704}]
W20250110 20:57:54.287415 7287 consensus_peers.cc:487] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Couldn't send request to peer 244d9ba396f549f2a5015792b6c5fff8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:54.290208 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 5a955851581349debfc895eed2411635 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.291283 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a955851581349debfc895eed2411635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.293135 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 5a955851581349debfc895eed2411635 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.294148 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a955851581349debfc895eed2411635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.293967 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 5a955851581349debfc895eed2411635 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.295243 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a955851581349debfc895eed2411635. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.298426 7617 ts_tablet_manager.cc:1918] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.298796 7617 log.cc:1198] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/7128e3c4f59141b0943f03ba7261ff04
I20250110 20:57:54.299439 7617 ts_tablet_manager.cc:1939] T 7128e3c4f59141b0943f03ba7261ff04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:54.302012 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 7128e3c4f59141b0943f03ba7261ff04 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:54.303732 7606 tablet_bootstrap.cc:492] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:54.304143 7606 ts_tablet_manager.cc:1397] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.000s
I20250110 20:57:54.306340 7606 raft_consensus.cc:357] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.307003 7606 raft_consensus.cc:383] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.307245 7606 raft_consensus.cc:738] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.311815 7616 raft_consensus.cc:2238] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.314061 7616 pending_rounds.cc:62] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Trying to abort 2 pending ops.
I20250110 20:57:54.314365 7616 pending_rounds.cc:69] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878793476296704 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "23b9983db2c649ea964e57a4dff784de" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_7a871347a918413297f7da917134a92e" dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" new_extra_config { } }
I20250110 20:57:54.315613 7616 pending_rounds.cc:69] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878792378056704 op_type: NO_OP noop_request { }
I20250110 20:57:54.316020 7616 raft_consensus.cc:2883] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:54.317766 7616 raft_consensus.cc:2267] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.311650 7609 tablet_bootstrap.cc:492] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:54.318529 7609 ts_tablet_manager.cc:1397] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.001s
I20250110 20:57:54.320631 7609 raft_consensus.cc:357] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:57:54.321367 7288 consensus_peers.cc:487] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Couldn't send request to peer c3b63d95ca734d8a9803b96bfd94f307. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:54.323014 7616 ts_tablet_manager.cc:1905] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.322786 7606 consensus_queue.cc:260] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.328285 7615 ts_tablet_manager.cc:1905] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.321257 7609 raft_consensus.cc:383] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.329595 7609 raft_consensus.cc:738] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.333431 7609 consensus_queue.cc:260] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.340473 7609 ts_tablet_manager.cc:1428] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20250110 20:57:54.346563 7616 ts_tablet_manager.cc:1918] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:54.346948 7616 log.cc:1198] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/23b9983db2c649ea964e57a4dff784de
I20250110 20:57:54.347630 7616 ts_tablet_manager.cc:1939] T 23b9983db2c649ea964e57a4dff784de P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:54.350282 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 23b9983db2c649ea964e57a4dff784de (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:54.356992 7609 tablet_bootstrap.cc:492] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:54.362516 7609 tablet_bootstrap.cc:654] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.374600 7606 ts_tablet_manager.cc:1428] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.070s user 0.006s sys 0.001s
I20250110 20:57:54.348951 7671 tablet.cc:1721] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:54.381605 7617 tablet_replica.cc:331] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:54.382481 7617 raft_consensus.cc:2238] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:54.383610 7617 raft_consensus.cc:2267] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.387825 7606 tablet_bootstrap.cc:492] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:54.393224 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet b6f6922d96c34c54b777160cd8d5c0d0 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:54.411293 7602 tablet_bootstrap.cc:492] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:54.416544 7602 tablet_bootstrap.cc:654] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:54.425194 7674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:57:54.427531 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet b6f6922d96c34c54b777160cd8d5c0d0
I20250110 20:57:54.429271 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f6922d96c34c54b777160cd8d5c0d0 with delete_type TABLET_DATA_DELETED (replaced by tablet f04e8ac3cccd4c3a984de3ddaa3d9187) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:54.432000 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 167292984bbc45faaabd68f0aee8212b from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:54.434814 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f6922d96c34c54b777160cd8d5c0d0 with delete_type TABLET_DATA_DELETED (replaced by tablet f04e8ac3cccd4c3a984de3ddaa3d9187) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:54.436959 7606 tablet_bootstrap.cc:654] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.437919 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet b6f6922d96c34c54b777160cd8d5c0d0 with delete_type TABLET_DATA_DELETED (replaced by tablet f04e8ac3cccd4c3a984de3ddaa3d9187) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:54.438647 7616 tablet_replica.cc:331] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:54.439392 7616 raft_consensus.cc:2238] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.439875 7616 raft_consensus.cc:2267] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.441555 7616 ts_tablet_manager.cc:1905] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.445601 7602 tablet_bootstrap.cc:492] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:54.446110 7602 ts_tablet_manager.cc:1397] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.004s
I20250110 20:57:54.454197 7617 ts_tablet_manager.cc:1905] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.454663 7609 tablet_bootstrap.cc:492] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:54.455052 7609 ts_tablet_manager.cc:1397] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.098s user 0.019s sys 0.038s
I20250110 20:57:54.456241 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 167292984bbc45faaabd68f0aee8212b
I20250110 20:57:54.457791 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet 167292984bbc45faaabd68f0aee8212b with delete_type TABLET_DATA_DELETED (replaced by tablet ddbadacad9ec42f98b2bd839aafcafa7) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:54.456977 7602 raft_consensus.cc:357] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.460296 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 167292984bbc45faaabd68f0aee8212b deleted and replaced by tablet ddbadacad9ec42f98b2bd839aafcafa7
I20250110 20:57:54.460469 7602 raft_consensus.cc:383] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.461294 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 167292984bbc45faaabd68f0aee8212b with delete_type TABLET_DATA_DELETED (replaced by tablet ddbadacad9ec42f98b2bd839aafcafa7) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:54.463222 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 167292984bbc45faaabd68f0aee8212b with delete_type TABLET_DATA_DELETED (replaced by tablet ddbadacad9ec42f98b2bd839aafcafa7) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:54.463613 7602 raft_consensus.cc:738] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.459164 7609 raft_consensus.cc:357] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.464558 7609 raft_consensus.cc:383] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.464829 7609 raft_consensus.cc:738] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.464670 7602 consensus_queue.cc:260] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.465492 7609 consensus_queue.cc:260] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.468621 6989 catalog_manager.cc:6596] ReplaceTablet: tablet b6f6922d96c34c54b777160cd8d5c0d0 deleted and replaced by tablet f04e8ac3cccd4c3a984de3ddaa3d9187
I20250110 20:57:54.474884 7616 ts_tablet_manager.cc:1918] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.475257 7616 log.cc:1198] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/b6f6922d96c34c54b777160cd8d5c0d0
I20250110 20:57:54.475889 7616 ts_tablet_manager.cc:1939] T b6f6922d96c34c54b777160cd8d5c0d0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:54.478076 7609 ts_tablet_manager.cc:1428] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250110 20:57:54.478963 7602 ts_tablet_manager.cc:1428] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20250110 20:57:54.480165 7616 tablet_replica.cc:331] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:54.480867 7616 raft_consensus.cc:2238] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.481292 7616 raft_consensus.cc:2267] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.482681 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet b6f6922d96c34c54b777160cd8d5c0d0 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:54.483000 7616 ts_tablet_manager.cc:1905] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.485318 7615 ts_tablet_manager.cc:1918] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:54.498147 7616 ts_tablet_manager.cc:1918] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.498499 7616 log.cc:1198] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/167292984bbc45faaabd68f0aee8212b
I20250110 20:57:54.499141 7616 ts_tablet_manager.cc:1939] T 167292984bbc45faaabd68f0aee8212b P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:54.501667 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 167292984bbc45faaabd68f0aee8212b (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:54.502249 7615 log.cc:1198] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/23b9983db2c649ea964e57a4dff784de
I20250110 20:57:54.505815 7615 ts_tablet_manager.cc:1939] T 23b9983db2c649ea964e57a4dff784de P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:54.492412 7602 tablet_bootstrap.cc:492] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:54.508459 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 23b9983db2c649ea964e57a4dff784de (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:54.520694 7615 tablet_replica.cc:331] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:54.521034 7602 tablet_bootstrap.cc:654] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.522161 6991 catalog_manager.cc:3422] renamed table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401] in HMS: new name default.table_59954cb851e943ff9f10bca64b86d8b8
I20250110 20:57:54.523741 7606 tablet_bootstrap.cc:492] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:54.524097 7615 raft_consensus.cc:2238] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:54.524207 7606 ts_tablet_manager.cc:1397] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.001s
I20250110 20:57:54.527794 7617 ts_tablet_manager.cc:1918] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:54.528885 6989 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:54.530385 6990 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:54.526750 7606 raft_consensus.cc:357] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:57:54.531978 6990 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 5a955851581349debfc895eed2411635 consensus state has no leader
I20250110 20:57:54.532431 7606 raft_consensus.cc:383] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.533257 7188 tablet_service.cc:1467] Processing CreateTablet for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.533887 7606 raft_consensus.cc:738] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.534443 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04e8ac3cccd4c3a984de3ddaa3d9187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.537676 6990 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:54.539085 7606 consensus_queue.cc:260] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.541754 7333 tablet_service.cc:1467] Processing CreateTablet for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.544572 7478 tablet_service.cc:1467] Processing CreateTablet for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.545553 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04e8ac3cccd4c3a984de3ddaa3d9187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.545564 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04e8ac3cccd4c3a984de3ddaa3d9187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.545871 7615 pending_rounds.cc:62] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:57:54.546139 7615 pending_rounds.cc:69] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878791972188160 op_type: NO_OP noop_request { }
I20250110 20:57:54.546514 7615 raft_consensus.cc:2883] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:54.546842 7615 raft_consensus.cc:2267] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.555289 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:54.561064 7615 ts_tablet_manager.cc:1905] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.564693 7606 ts_tablet_manager.cc:1428] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.040s user 0.004s sys 0.000s
I20250110 20:57:54.564729 7190 tablet_service.cc:1467] Processing CreateTablet for tablet ddbadacad9ec42f98b2bd839aafcafa7 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.567206 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbadacad9ec42f98b2bd839aafcafa7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:54.570462 6980 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5a955851581349debfc895eed2411635 consensus state has no leader
I20250110 20:57:54.571275 7477 tablet_service.cc:1467] Processing CreateTablet for tablet ddbadacad9ec42f98b2bd839aafcafa7 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.572335 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbadacad9ec42f98b2bd839aafcafa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.566437 7617 log.cc:1198] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/23b9983db2c649ea964e57a4dff784de
I20250110 20:57:54.567051 7332 tablet_service.cc:1467] Processing CreateTablet for tablet ddbadacad9ec42f98b2bd839aafcafa7 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.580843 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ddbadacad9ec42f98b2bd839aafcafa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.567292 6990 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:54.589949 7606 tablet_bootstrap.cc:492] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:54.594228 6989 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:54.595199 7606 tablet_bootstrap.cc:654] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.601622 7602 tablet_bootstrap.cc:492] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:54.602005 7609 tablet_bootstrap.cc:492] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:54.602149 7602 ts_tablet_manager.cc:1397] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.110s user 0.009s sys 0.000s
I20250110 20:57:54.604347 7602 raft_consensus.cc:357] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.605085 7602 raft_consensus.cc:383] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.605376 7602 raft_consensus.cc:738] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.606132 7602 consensus_queue.cc:260] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.607100 7609 tablet_bootstrap.cc:654] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.595232 7617 ts_tablet_manager.cc:1939] T 23b9983db2c649ea964e57a4dff784de P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:54.612038 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 23b9983db2c649ea964e57a4dff784de (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:54.613529 7606 tablet_bootstrap.cc:492] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:54.613888 7606 ts_tablet_manager.cc:1397] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.001s
I20250110 20:57:54.614437 7617 tablet_replica.cc:331] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:54.615211 7617 raft_consensus.cc:2238] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.618405 7609 tablet_bootstrap.cc:492] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:54.618763 7609 ts_tablet_manager.cc:1397] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20250110 20:57:54.620779 7609 raft_consensus.cc:357] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.621287 7609 raft_consensus.cc:383] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.621519 7609 raft_consensus.cc:738] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.622071 7609 consensus_queue.cc:260] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.624076 7609 ts_tablet_manager.cc:1428] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20250110 20:57:54.624837 7609 tablet_bootstrap.cc:492] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:54.630302 7609 tablet_bootstrap.cc:654] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.641456 7602 ts_tablet_manager.cc:1428] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.039s user 0.001s sys 0.003s
I20250110 20:57:54.615936 7606 raft_consensus.cc:357] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.641928 7606 raft_consensus.cc:383] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.642228 7606 raft_consensus.cc:738] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.643290 7606 consensus_queue.cc:260] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.647356 7606 ts_tablet_manager.cc:1428] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.033s user 0.005s sys 0.001s
I20250110 20:57:54.648137 7606 tablet_bootstrap.cc:492] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:54.659986 7606 tablet_bootstrap.cc:654] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.663106 7602 tablet_bootstrap.cc:492] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:54.671886 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 3c414aabf453406fb42216a6d4121882 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:54.675527 7609 tablet_bootstrap.cc:492] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:54.675904 7609 ts_tablet_manager.cc:1397] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.051s user 0.013s sys 0.002s
I20250110 20:57:54.642283 7617 raft_consensus.cc:2267] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.678318 7609 raft_consensus.cc:357] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.678946 7609 raft_consensus.cc:383] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.679203 7609 raft_consensus.cc:738] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.680040 7617 ts_tablet_manager.cc:1905] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.679744 7609 consensus_queue.cc:260] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.684527 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet f04e8ac3cccd4c3a984de3ddaa3d9187 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:54.688032 7602 tablet_bootstrap.cc:654] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.689918 7609 ts_tablet_manager.cc:1428] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250110 20:57:54.693603 6987 catalog_manager.cc:2149] created HMS catalog entry for table [id=c4582c5662964616b35896bcce266376]
I20250110 20:57:54.701892 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 3c414aabf453406fb42216a6d4121882
I20250110 20:57:54.703637 6989 catalog_manager.cc:6596] ReplaceTablet: tablet 3c414aabf453406fb42216a6d4121882 deleted and replaced by tablet d7f69b85ab3e4b2587d5461506533067
I20250110 20:57:54.709497 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c414aabf453406fb42216a6d4121882 with delete_type TABLET_DATA_DELETED (replaced by tablet d7f69b85ab3e4b2587d5461506533067) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:54.711087 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c414aabf453406fb42216a6d4121882 with delete_type TABLET_DATA_DELETED (replaced by tablet d7f69b85ab3e4b2587d5461506533067) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:54.711833 7616 tablet_replica.cc:331] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:54.712602 7616 raft_consensus.cc:2238] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.713073 7616 raft_consensus.cc:2267] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.703467 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c414aabf453406fb42216a6d4121882 with delete_type TABLET_DATA_DELETED (replaced by tablet d7f69b85ab3e4b2587d5461506533067) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:54.715212 7616 ts_tablet_manager.cc:1905] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.730918 7615 ts_tablet_manager.cc:1918] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:54.731357 7615 log.cc:1198] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/b6f6922d96c34c54b777160cd8d5c0d0
I20250110 20:57:54.732156 7615 ts_tablet_manager.cc:1939] T b6f6922d96c34c54b777160cd8d5c0d0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:54.734953 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet b6f6922d96c34c54b777160cd8d5c0d0 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
W20250110 20:57:54.739950 7288 consensus_peers.cc:487] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Couldn't send request to peer c3b63d95ca734d8a9803b96bfd94f307. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 167292984bbc45faaabd68f0aee8212b. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:54.743211 7617 ts_tablet_manager.cc:1918] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.753646 7610 raft_consensus.cc:491] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:54.754487 7610 raft_consensus.cc:513] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.755360 7615 tablet_replica.cc:331] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:54.756333 7610 leader_election.cc:290] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:54.757778 7615 raft_consensus.cc:2238] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.758245 7615 raft_consensus.cc:2267] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.758927 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04e8ac3cccd4c3a984de3ddaa3d9187" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:54.760165 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04e8ac3cccd4c3a984de3ddaa3d9187" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:54.760794 7501 raft_consensus.cc:2463] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:54.760732 6987 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 10) took 1769 ms. Trace:
W20250110 20:57:54.762279 7287 leader_election.cc:343] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:54.763434 7288 leader_election.cc:304] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:57:54.764542 7610 raft_consensus.cc:2798] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:54.764859 7610 raft_consensus.cc:491] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:54.761170 6987 rpcz_store.cc:276] 0110 20:57:52.991124 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:52.993626 (+ 2502us) service_pool.cc:225] Handling call
0110 20:57:54.138978 (+1145352us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:54.140786 (+ 1808us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:54.693558 (+552772us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:57:54.760084 (+ 66526us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:54.760405 (+ 321us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:54.760670 (+ 265us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:54.731654 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:54.732102 (+ 448us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:54.732128 (+ 26us) write_op.cc:435] Acquired schema lock
0110 20:57:54.732147 (+ 19us) tablet.cc:659] Decoding operations
0110 20:57:54.732432 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:54.732470 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:54.732498 (+ 28us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:54.732703 (+ 205us) tablet.cc:698] Row locks acquired
0110 20:57:54.732716 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:54.732845 (+ 129us) write_op.cc:270] Start()
0110 20:57:54.732936 (+ 91us) write_op.cc:276] Timestamp: P: 1736542674732769 usec, L: 0
0110 20:57:54.732955 (+ 19us) op_driver.cc:348] REPLICATION: starting
0110 20:57:54.736002 (+ 3047us) log.cc:844] Serialized 432 byte log entry
0110 20:57:54.758326 (+ 22324us) op_driver.cc:464] REPLICATION: finished
0110 20:57:54.758752 (+ 426us) write_op.cc:301] APPLY: starting
0110 20:57:54.758798 (+ 46us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:54.758902 (+ 104us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:54.758984 (+ 82us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:54.759315 (+ 331us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:54.759441 (+ 126us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:54.759480 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:57:54.759640 (+ 160us) log.cc:844] Serialized 28 byte log entry
0110 20:57:54.759764 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:54.759842 (+ 78us) write_op.cc:454] Released schema lock
0110 20:57:54.759870 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:54.760104 (+ 234us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:54.760126 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":380284,"Hive Metastore.run_cpu_time_us":557,"Hive Metastore.run_wall_time_us":171558,"child_traces":[["op",{"apply.queue_time_us":286,"apply.run_cpu_time_us":1319,"apply.run_wall_time_us":1318,"mutex_wait_us":34918,"num_ops":2,"prepare.queue_time_us":36530,"prepare.run_cpu_time_us":2144,"prepare.run_wall_time_us":5745,"raft.queue_time_us":78,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":18,"replication_time_us":25287}]]}
I20250110 20:57:54.765169 7610 raft_consensus.cc:3054] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:54.766459 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet f04e8ac3cccd4c3a984de3ddaa3d9187
I20250110 20:57:54.768700 6990 catalog_manager.cc:6596] ReplaceTablet: tablet f04e8ac3cccd4c3a984de3ddaa3d9187 deleted and replaced by tablet 89212badb30649dfb65e51c57760f787
I20250110 20:57:54.770380 7616 ts_tablet_manager.cc:1918] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.770774 7616 log.cc:1198] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/3c414aabf453406fb42216a6d4121882
I20250110 20:57:54.771421 7616 ts_tablet_manager.cc:1939] T 3c414aabf453406fb42216a6d4121882 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:54.771814 7617 log.cc:1198] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/b6f6922d96c34c54b777160cd8d5c0d0
I20250110 20:57:54.773429 7602 tablet_bootstrap.cc:492] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:54.768018 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 with delete_type TABLET_DATA_DELETED (replaced by tablet 89212badb30649dfb65e51c57760f787) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:54.774319 7602 ts_tablet_manager.cc:1397] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.005s
I20250110 20:57:54.776535 7602 raft_consensus.cc:357] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.778507 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 with delete_type TABLET_DATA_DELETED (replaced by tablet 89212badb30649dfb65e51c57760f787) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:54.778926 7602 raft_consensus.cc:383] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.778697 7617 ts_tablet_manager.cc:1939] T b6f6922d96c34c54b777160cd8d5c0d0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:54.780184 7606 tablet_bootstrap.cc:492] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:54.759948 7615 ts_tablet_manager.cc:1905] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.780557 7606 ts_tablet_manager.cc:1397] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.133s user 0.024s sys 0.031s
I20250110 20:57:54.777604 7610 raft_consensus.cc:513] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.783233 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04e8ac3cccd4c3a984de3ddaa3d9187" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:54.783383 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 with delete_type TABLET_DATA_DELETED (replaced by tablet 89212badb30649dfb65e51c57760f787) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:54.784567 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 with delete_type TABLET_DATA_DELETED (replaced by tablet 89212badb30649dfb65e51c57760f787) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:54.780534 7602 raft_consensus.cc:738] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.797044 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 3c414aabf453406fb42216a6d4121882 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:54.797349 7616 tablet_replica.cc:331] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:54.798108 7616 raft_consensus.cc:2238] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.799580 7606 raft_consensus.cc:357] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.799902 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet b6f6922d96c34c54b777160cd8d5c0d0 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:54.800262 7606 raft_consensus.cc:383] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.800501 7606 raft_consensus.cc:738] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.801059 7606 consensus_queue.cc:260] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.801546 7616 raft_consensus.cc:2267] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.802232 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04e8ac3cccd4c3a984de3ddaa3d9187" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:54.804035 7606 ts_tablet_manager.cc:1428] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20250110 20:57:54.804795 7616 ts_tablet_manager.cc:1905] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:54.808918 7288 leader_election.cc:343] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:54.813604 7190 tablet_service.cc:1467] Processing CreateTablet for tablet d7f69b85ab3e4b2587d5461506533067 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.814669 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f69b85ab3e4b2587d5461506533067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.817613 7477 tablet_service.cc:1467] Processing CreateTablet for tablet d7f69b85ab3e4b2587d5461506533067 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.818639 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f69b85ab3e4b2587d5461506533067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.819460 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 89212badb30649dfb65e51c57760f787 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.820443 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89212badb30649dfb65e51c57760f787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.817006 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 118976382cf440eebb9d00973928491f (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.822031 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 118976382cf440eebb9d00973928491f (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.823029 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118976382cf440eebb9d00973928491f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:54.831305 7436 consensus_peers.cc:487] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:54.832163 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118976382cf440eebb9d00973928491f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.836247 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 89212badb30649dfb65e51c57760f787 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.837332 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89212badb30649dfb65e51c57760f787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.841223 7609 tablet_bootstrap.cc:492] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:54.845129 7609 tablet_bootstrap.cc:654] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.845268 7332 tablet_service.cc:1467] Processing CreateTablet for tablet d7f69b85ab3e4b2587d5461506533067 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.846367 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f69b85ab3e4b2587d5461506533067. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.850589 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 118976382cf440eebb9d00973928491f (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.850890 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 89212badb30649dfb65e51c57760f787 (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:54.851614 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 118976382cf440eebb9d00973928491f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.852133 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89212badb30649dfb65e51c57760f787. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:54.859308 7639 consensus_queue.cc:1035] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:54.861853 7615 ts_tablet_manager.cc:1918] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.862208 7615 log.cc:1198] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/167292984bbc45faaabd68f0aee8212b
I20250110 20:57:54.862826 7615 ts_tablet_manager.cc:1939] T 167292984bbc45faaabd68f0aee8212b P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:54.798008 7602 consensus_queue.cc:260] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.863615 7210 raft_consensus.cc:3054] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:54.872989 7210 raft_consensus.cc:2463] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:54.874670 7606 tablet_bootstrap.cc:492] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
W20250110 20:57:54.874620 7436 consensus_peers.cc:487] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:57:54.878029 7287 leader_election.cc:304] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:54.878894 7646 raft_consensus.cc:2798] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:54.881577 7606 tablet_bootstrap.cc:654] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.882165 7689 raft_consensus.cc:491] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:54.885443 7639 consensus_queue.cc:1035] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:54.885442 7610 leader_election.cc:290] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:54.892585 7615 tablet_replica.cc:331] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:54.892725 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 167292984bbc45faaabd68f0aee8212b (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:54.893352 7615 raft_consensus.cc:2238] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:54.893865 7615 raft_consensus.cc:2267] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.895172 7617 tablet_replica.cc:331] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:54.895933 7617 raft_consensus.cc:2238] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:54.900703 7602 ts_tablet_manager.cc:1428] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.126s user 0.004s sys 0.001s
I20250110 20:57:54.882593 7689 raft_consensus.cc:513] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.909900 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c414aabf453406fb42216a6d4121882" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
W20250110 20:57:54.911015 7287 leader_election.cc:343] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:54.911101 7689 leader_election.cc:290] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:57:54.917439 7288 leader_election.cc:343] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 3c414aabf453406fb42216a6d4121882
I20250110 20:57:54.917888 7288 leader_election.cc:304] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:54.920774 7606 tablet_bootstrap.cc:492] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:54.921185 7606 ts_tablet_manager.cc:1397] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.005s
I20250110 20:57:54.921648 7602 tablet_bootstrap.cc:492] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:54.921551 7689 raft_consensus.cc:2743] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:54.924160 7606 raft_consensus.cc:357] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.925436 7606 raft_consensus.cc:383] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.925686 7606 raft_consensus.cc:738] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:54.926302 7606 consensus_queue.cc:260] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.926724 7646 raft_consensus.cc:695] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:54.928243 7606 ts_tablet_manager.cc:1428] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250110 20:57:54.927538 7646 consensus_queue.cc:237] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:57:54.935753 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:54.935899 7616 ts_tablet_manager.cc:1918] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:54.936236 7616 log.cc:1198] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/f04e8ac3cccd4c3a984de3ddaa3d9187
I20250110 20:57:54.936796 7616 ts_tablet_manager.cc:1939] T f04e8ac3cccd4c3a984de3ddaa3d9187 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:54.896672 7617 pending_rounds.cc:62] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:57:54.937492 7617 pending_rounds.cc:69] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878792356016128 op_type: NO_OP noop_request { }
I20250110 20:57:54.937904 7617 raft_consensus.cc:2883] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:54.938242 7617 raft_consensus.cc:2267] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:54.938810 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet f04e8ac3cccd4c3a984de3ddaa3d9187 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
W20250110 20:57:54.947280 6980 catalog_manager.cc:4839] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet f04e8ac3cccd4c3a984de3ddaa3d9187 because the tablet was not found. No further retry: Not found: Tablet not found: f04e8ac3cccd4c3a984de3ddaa3d9187
I20250110 20:57:54.947887 7606 tablet_bootstrap.cc:492] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
W20250110 20:57:54.948926 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:54.956630 7606 tablet_bootstrap.cc:654] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:54.960264 7617 ts_tablet_manager.cc:1905] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.966794 7615 ts_tablet_manager.cc:1905] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:54.969223 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 3c414aabf453406fb42216a6d4121882 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:54.970422 6987 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_dca3421fe79d4330bc146c208b21a6f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:54.981017 7609 tablet_bootstrap.cc:492] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:54.981443 7609 ts_tablet_manager.cc:1397] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.140s user 0.013s sys 0.003s
I20250110 20:57:54.985394 7609 raft_consensus.cc:357] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:54.986075 7609 raft_consensus.cc:383] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:54.987537 7602 tablet_bootstrap.cc:654] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:55.003185 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:57:55.011101 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:57:55.045673 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet f04e8ac3cccd4c3a984de3ddaa3d9187
I20250110 20:57:55.037389 7617 ts_tablet_manager.cc:1918] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:55.055949 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 5a955851581349debfc895eed2411635 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:55.056411 7617 log.cc:1198] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/167292984bbc45faaabd68f0aee8212b
I20250110 20:57:55.057148 7617 ts_tablet_manager.cc:1939] T 167292984bbc45faaabd68f0aee8212b P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
W20250110 20:57:55.058735 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:55.059789 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 167292984bbc45faaabd68f0aee8212b (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:55.042268 7615 ts_tablet_manager.cc:1918] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.062103 7615 log.cc:1198] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/3c414aabf453406fb42216a6d4121882
I20250110 20:57:54.986315 7609 raft_consensus.cc:738] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.061244 7602 tablet_bootstrap.cc:492] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:55.063711 7602 ts_tablet_manager.cc:1397] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.162s user 0.011s sys 0.000s
I20250110 20:57:55.070179 7609 consensus_queue.cc:260] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.072008 7602 raft_consensus.cc:357] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.072623 7602 raft_consensus.cc:383] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.076635 7611 raft_consensus.cc:491] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:55.077050 7611 raft_consensus.cc:513] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.078824 7611 leader_election.cc:290] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.079633 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 5a955851581349debfc895eed2411635
I20250110 20:57:55.062724 7615 ts_tablet_manager.cc:1939] T 3c414aabf453406fb42216a6d4121882 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:55.080406 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a955851581349debfc895eed2411635" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:55.080996 7353 raft_consensus.cc:2463] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:55.081741 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a955851581349debfc895eed2411635 with delete_type TABLET_DATA_DELETED (replaced by tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:55.081768 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a955851581349debfc895eed2411635 with delete_type TABLET_DATA_DELETED (replaced by tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:55.072861 7602 raft_consensus.cc:738] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.083297 7145 leader_election.cc:304] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:55.084121 7617 tablet_replica.cc:331] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:55.084548 7602 consensus_queue.cc:260] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:57:55.086918 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:55.087222 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 5a955851581349debfc895eed2411635 deleted and replaced by tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025
I20250110 20:57:55.090464 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 3c414aabf453406fb42216a6d4121882 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:55.091013 7617 raft_consensus.cc:2238] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.091440 7617 raft_consensus.cc:2267] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.093145 7617 ts_tablet_manager.cc:1905] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.094511 7606 tablet_bootstrap.cc:492] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:55.094919 7606 ts_tablet_manager.cc:1397] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.148s user 0.009s sys 0.009s
I20250110 20:57:55.097023 7606 raft_consensus.cc:357] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.097892 7606 raft_consensus.cc:383] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.098191 7606 raft_consensus.cc:738] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.085906 7611 raft_consensus.cc:2798] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:55.098860 7611 raft_consensus.cc:491] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:55.099197 7611 raft_consensus.cc:3054] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.098809 7606 consensus_queue.cc:260] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.080658 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a955851581349debfc895eed2411635" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:55.103991 7501 raft_consensus.cc:2463] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:55.082760 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a955851581349debfc895eed2411635 with delete_type TABLET_DATA_DELETED (replaced by tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:55.105057 7616 tablet_replica.cc:331] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:55.105698 7616 raft_consensus.cc:2238] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.106099 7616 raft_consensus.cc:2267] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.107599 7616 ts_tablet_manager.cc:1905] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.114504 7615 tablet_replica.cc:331] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:55.115382 7615 raft_consensus.cc:2238] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.116245 7615 raft_consensus.cc:2267] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.113163 7611 raft_consensus.cc:513] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.124073 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a955851581349debfc895eed2411635" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:55.124610 7353 raft_consensus.cc:3054] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.125397 7332 tablet_service.cc:1467] Processing CreateTablet for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.126468 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.126616 7190 tablet_service.cc:1467] Processing CreateTablet for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.127673 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.130632 7481 tablet_service.cc:1467] Processing CreateTablet for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.131634 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.139396 7353 raft_consensus.cc:2463] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:55.140702 7145 leader_election.cc:304] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:55.147768 7611 leader_election.cc:290] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.147912 7616 ts_tablet_manager.cc:1918] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.148243 7616 log.cc:1198] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/5a955851581349debfc895eed2411635
I20250110 20:57:55.148511 7611 raft_consensus.cc:2798] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:55.148798 7616 ts_tablet_manager.cc:1939] T 5a955851581349debfc895eed2411635 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:55.148896 7611 raft_consensus.cc:695] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:57:55.149578 7611 consensus_queue.cc:237] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:57:55.152173 6980 consensus_peers.cc:487] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 -> Peer f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203): Couldn't send request to peer f9bb61bdf30044a1b482d6a66e29435f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250110 20:57:55.153862 7602 ts_tablet_manager.cc:1428] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.090s user 0.008s sys 0.000s
I20250110 20:57:55.158017 7615 ts_tablet_manager.cc:1905] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.158936 7602 tablet_bootstrap.cc:492] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:55.160348 7609 ts_tablet_manager.cc:1428] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.179s user 0.005s sys 0.000s
I20250110 20:57:55.160940 7606 ts_tablet_manager.cc:1428] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.066s user 0.003s sys 0.002s
I20250110 20:57:55.165026 7602 tablet_bootstrap.cc:654] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.165992 7606 tablet_bootstrap.cc:492] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:55.182536 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:55.185892 7609 tablet_bootstrap.cc:492] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:55.196347 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet d7f69b85ab3e4b2587d5461506533067 from {username='slave'} at 127.0.0.1:33774
W20250110 20:57:55.198132 7145 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.194:34035, user_credentials={real_user=slave}} blocked reactor thread for 57513.6us
W20250110 20:57:55.204061 7145 leader_election.cc:343] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 5a955851581349debfc895eed2411635
I20250110 20:57:55.216900 7611 raft_consensus.cc:491] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:55.217325 7611 raft_consensus.cc:513] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.219184 7611 leader_election.cc:290] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.220641 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbadacad9ec42f98b2bd839aafcafa7" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:55.221279 7353 raft_consensus.cc:2463] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:55.222716 7145 leader_election.cc:304] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:55.223217 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet d7f69b85ab3e4b2587d5461506533067
I20250110 20:57:55.223592 7611 raft_consensus.cc:2798] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:55.223949 7611 raft_consensus.cc:491] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:55.224459 6990 catalog_manager.cc:6596] ReplaceTablet: tablet d7f69b85ab3e4b2587d5461506533067 deleted and replaced by tablet a632c3126b01431abf5772f192753040
I20250110 20:57:55.224282 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbadacad9ec42f98b2bd839aafcafa7" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:55.224897 7501 raft_consensus.cc:2463] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:55.225283 7611 raft_consensus.cc:3054] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.227242 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet d7f69b85ab3e4b2587d5461506533067 with delete_type TABLET_DATA_DELETED (replaced by tablet a632c3126b01431abf5772f192753040) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:55.227484 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet d7f69b85ab3e4b2587d5461506533067 with delete_type TABLET_DATA_DELETED (replaced by tablet a632c3126b01431abf5772f192753040) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:55.230057 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet d7f69b85ab3e4b2587d5461506533067 with delete_type TABLET_DATA_DELETED (replaced by tablet a632c3126b01431abf5772f192753040) from {username='slave'} at 127.0.0.1:43396
W20250110 20:57:55.231829 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet d7f69b85ab3e4b2587d5461506533067 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7f69b85ab3e4b2587d5461506533067 already in progress: creating tablet
I20250110 20:57:55.250556 7602 tablet_bootstrap.cc:492] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:55.251052 7602 ts_tablet_manager.cc:1397] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.000s
I20250110 20:57:55.259434 7615 ts_tablet_manager.cc:1918] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.260242 7615 log.cc:1198] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/f04e8ac3cccd4c3a984de3ddaa3d9187
I20250110 20:57:55.260883 7615 ts_tablet_manager.cc:1939] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:55.261632 7606 tablet_bootstrap.cc:654] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.264147 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet f04e8ac3cccd4c3a984de3ddaa3d9187 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:55.264400 7615 tablet_replica.cc:331] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:55.265039 7615 raft_consensus.cc:2238] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:55.265609 7615 pending_rounds.cc:62] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:57:55.265851 7615 pending_rounds.cc:69] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878797496283136 op_type: NO_OP noop_request { }
I20250110 20:57:55.266180 7615 raft_consensus.cc:2883] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:55.266458 7615 raft_consensus.cc:2267] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.269769 7190 tablet_service.cc:1467] Processing CreateTablet for tablet a632c3126b01431abf5772f192753040 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.270898 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a632c3126b01431abf5772f192753040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.271471 7332 tablet_service.cc:1467] Processing CreateTablet for tablet a632c3126b01431abf5772f192753040 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.272734 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a632c3126b01431abf5772f192753040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.272588 7611 raft_consensus.cc:513] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.273669 7609 tablet_bootstrap.cc:654] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.274930 7481 tablet_service.cc:1467] Processing CreateTablet for tablet a632c3126b01431abf5772f192753040 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.275996 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a632c3126b01431abf5772f192753040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.281468 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbadacad9ec42f98b2bd839aafcafa7" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:55.282076 7353 raft_consensus.cc:3054] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.283138 7611 leader_election.cc:290] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.257360 7602 raft_consensus.cc:357] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.283721 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbadacad9ec42f98b2bd839aafcafa7" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:55.284154 7602 raft_consensus.cc:383] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.284260 7501 raft_consensus.cc:3054] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.284394 7602 raft_consensus.cc:738] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.284937 7602 consensus_queue.cc:260] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.286885 7602 ts_tablet_manager.cc:1428] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.036s user 0.002s sys 0.003s
I20250110 20:57:55.288493 7353 raft_consensus.cc:2463] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:55.293670 7615 ts_tablet_manager.cc:1905] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.294001 7145 leader_election.cc:304] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:55.294777 7611 raft_consensus.cc:2798] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:55.295495 7611 raft_consensus.cc:695] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:57:55.296243 7611 consensus_queue.cc:237] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.303568 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:55.307592 7501 raft_consensus.cc:2463] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:55.308307 7609 tablet_bootstrap.cc:492] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:55.308696 7609 ts_tablet_manager.cc:1397] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.000s
I20250110 20:57:55.312191 7609 raft_consensus.cc:357] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.312810 7609 raft_consensus.cc:383] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.313050 7609 raft_consensus.cc:738] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.313642 7609 consensus_queue.cc:260] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.315410 7617 ts_tablet_manager.cc:1918] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.315559 7609 ts_tablet_manager.cc:1428] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20250110 20:57:55.315763 7617 log.cc:1198] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/3c414aabf453406fb42216a6d4121882
I20250110 20:57:55.316331 7609 tablet_bootstrap.cc:492] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:55.316367 7617 ts_tablet_manager.cc:1939] T 3c414aabf453406fb42216a6d4121882 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:55.318771 7602 tablet_bootstrap.cc:492] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:55.333820 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet d7f69b85ab3e4b2587d5461506533067 with delete_type TABLET_DATA_DELETED (replaced by tablet a632c3126b01431abf5772f192753040) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:55.334565 7616 tablet_replica.cc:331] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:55.335219 7616 raft_consensus.cc:2238] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.335631 7616 raft_consensus.cc:2267] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.335655 7602 tablet_bootstrap.cc:654] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.337216 7616 ts_tablet_manager.cc:1905] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.339352 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 3c414aabf453406fb42216a6d4121882 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:55.348816 7606 tablet_bootstrap.cc:492] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:55.349211 7606 ts_tablet_manager.cc:1397] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.183s user 0.007s sys 0.007s
I20250110 20:57:55.351521 7606 raft_consensus.cc:357] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.352226 7606 raft_consensus.cc:383] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.352483 7606 raft_consensus.cc:738] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.356357 6990 catalog_manager.cc:5526] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:55.357241 7617 tablet_replica.cc:331] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:55.357911 7617 raft_consensus.cc:2238] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:55.358556 7617 pending_rounds.cc:62] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:57:55.358769 7617 pending_rounds.cc:69] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878796529487872 op_type: NO_OP noop_request { }
I20250110 20:57:55.359115 7617 raft_consensus.cc:2883] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:55.359422 7617 raft_consensus.cc:2267] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.362687 7617 ts_tablet_manager.cc:1905] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.353109 7606 consensus_queue.cc:260] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.371222 7616 ts_tablet_manager.cc:1918] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.371583 7616 log.cc:1198] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/d7f69b85ab3e4b2587d5461506533067
I20250110 20:57:55.368357 7606 ts_tablet_manager.cc:1428] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250110 20:57:55.373618 7606 tablet_bootstrap.cc:492] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:55.376665 7616 ts_tablet_manager.cc:1939] T d7f69b85ab3e4b2587d5461506533067 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:55.378928 7609 tablet_bootstrap.cc:654] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.382205 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet d7f69b85ab3e4b2587d5461506533067 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:55.392690 7615 ts_tablet_manager.cc:1918] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:55.393091 7615 log.cc:1198] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/5a955851581349debfc895eed2411635
I20250110 20:57:55.393790 7615 ts_tablet_manager.cc:1939] T 5a955851581349debfc895eed2411635 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:55.406984 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:55.407265 7615 tablet_replica.cc:331] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:55.407944 7615 raft_consensus.cc:2238] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.408375 7615 raft_consensus.cc:2267] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.410429 7602 tablet_bootstrap.cc:492] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:55.411495 7602 ts_tablet_manager.cc:1397] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.003s
I20250110 20:57:55.413793 7602 raft_consensus.cc:357] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.414455 7602 raft_consensus.cc:383] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.414716 7602 raft_consensus.cc:738] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.415277 7602 consensus_queue.cc:260] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.417176 7602 ts_tablet_manager.cc:1428] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250110 20:57:55.418613 7697 raft_consensus.cc:491] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:55.419059 7697 raft_consensus.cc:513] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.420846 7697 leader_election.cc:290] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:55.423808 7606 tablet_bootstrap.cc:654] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.423568 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118976382cf440eebb9d00973928491f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:55.424240 7353 raft_consensus.cc:2463] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:55.439849 7436 leader_election.cc:304] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:55.440694 7697 raft_consensus.cc:2798] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:55.441043 7697 raft_consensus.cc:491] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:55.441350 7697 raft_consensus.cc:3054] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.447793 7697 raft_consensus.cc:513] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.455989 7615 ts_tablet_manager.cc:1905] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.467240 7609 tablet_bootstrap.cc:492] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:55.467646 7609 ts_tablet_manager.cc:1397] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.151s user 0.021s sys 0.023s
I20250110 20:57:55.470678 7697 leader_election.cc:290] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:55.471303 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118976382cf440eebb9d00973928491f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:55.471870 7353 raft_consensus.cc:3054] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.471848 7609 raft_consensus.cc:357] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.472539 7609 raft_consensus.cc:383] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.472767 7609 raft_consensus.cc:738] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.474190 7602 tablet_bootstrap.cc:492] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:55.476644 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118976382cf440eebb9d00973928491f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:55.477231 7208 raft_consensus.cc:3054] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.479622 7353 raft_consensus.cc:2463] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:55.480597 7436 leader_election.cc:304] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:55.483461 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118976382cf440eebb9d00973928491f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:55.489018 7697 raft_consensus.cc:2798] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:55.489456 7697 raft_consensus.cc:695] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:55.490204 7697 consensus_queue.cc:237] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.494596 7208 raft_consensus.cc:2463] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:55.496599 7602 tablet_bootstrap.cc:654] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.498013 7606 tablet_bootstrap.cc:492] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:55.498517 7606 ts_tablet_manager.cc:1397] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.126s user 0.009s sys 0.001s
I20250110 20:57:55.500973 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet ddbadacad9ec42f98b2bd839aafcafa7 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:55.500771 7606 raft_consensus.cc:357] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.500825 7609 consensus_queue.cc:260] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.504089 7609 ts_tablet_manager.cc:1428] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
I20250110 20:57:55.504390 7606 raft_consensus.cc:383] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.504688 7606 raft_consensus.cc:738] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.505445 7606 consensus_queue.cc:260] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.504866 7609 tablet_bootstrap.cc:492] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:55.514621 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 89212badb30649dfb65e51c57760f787 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:55.506752 7617 ts_tablet_manager.cc:1918] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:55.517576 7617 log.cc:1198] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f04e8ac3cccd4c3a984de3ddaa3d9187
I20250110 20:57:55.518257 7617 ts_tablet_manager.cc:1939] T f04e8ac3cccd4c3a984de3ddaa3d9187 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:55.520294 7617 tablet_replica.cc:331] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:55.528143 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet f04e8ac3cccd4c3a984de3ddaa3d9187 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:55.537747 7606 ts_tablet_manager.cc:1428] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250110 20:57:55.538691 7606 tablet_bootstrap.cc:492] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:55.540105 7617 raft_consensus.cc:2238] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.540496 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet ddbadacad9ec42f98b2bd839aafcafa7
I20250110 20:57:55.540534 7617 raft_consensus.cc:2267] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.542107 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbadacad9ec42f98b2bd839aafcafa7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d49b406d2e3438594c4d837422eb707) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:55.543355 7617 ts_tablet_manager.cc:1905] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.552563 7615 ts_tablet_manager.cc:1918] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.552961 7615 log.cc:1198] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/d7f69b85ab3e4b2587d5461506533067
I20250110 20:57:55.554725 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 89212badb30649dfb65e51c57760f787
I20250110 20:57:55.556731 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 89212badb30649dfb65e51c57760f787 with delete_type TABLET_DATA_DELETED (replaced by tablet 30aa8fd175de4c3aac137ac51f1ae51d) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:55.560640 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 89212badb30649dfb65e51c57760f787 with delete_type TABLET_DATA_DELETED (replaced by tablet 30aa8fd175de4c3aac137ac51f1ae51d) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:55.561337 7616 tablet_replica.cc:331] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:55.547830 7606 tablet_bootstrap.cc:654] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.562032 7616 raft_consensus.cc:2238] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.562450 7616 raft_consensus.cc:2267] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.561527 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 89212badb30649dfb65e51c57760f787 with delete_type TABLET_DATA_DELETED (replaced by tablet 30aa8fd175de4c3aac137ac51f1ae51d) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:55.564239 7616 ts_tablet_manager.cc:1905] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.566155 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbadacad9ec42f98b2bd839aafcafa7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d49b406d2e3438594c4d837422eb707) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:55.559526 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 89212badb30649dfb65e51c57760f787 deleted and replaced by tablet 30aa8fd175de4c3aac137ac51f1ae51d
I20250110 20:57:55.565129 6989 catalog_manager.cc:6596] ReplaceTablet: tablet ddbadacad9ec42f98b2bd839aafcafa7 deleted and replaced by tablet 7d49b406d2e3438594c4d837422eb707
I20250110 20:57:55.569006 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbadacad9ec42f98b2bd839aafcafa7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d49b406d2e3438594c4d837422eb707) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:55.582701 7646 raft_consensus.cc:491] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:55.583236 7646 raft_consensus.cc:513] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.585253 7646 leader_election.cc:290] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.586136 7615 ts_tablet_manager.cc:1939] T d7f69b85ab3e4b2587d5461506533067 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:55.586810 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7f69b85ab3e4b2587d5461506533067" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
W20250110 20:57:55.589618 7287 leader_election.cc:343] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: Tablet not RUNNING: SHUTDOWN
W20250110 20:57:55.590348 7288 leader_election.cc:343] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: d7f69b85ab3e4b2587d5461506533067
I20250110 20:57:55.590714 7288 leader_election.cc:304] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:55.591254 7602 tablet_bootstrap.cc:492] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:55.591627 7602 ts_tablet_manager.cc:1397] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.118s user 0.002s sys 0.011s
I20250110 20:57:55.591737 7646 raft_consensus.cc:2743] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:55.601647 7615 tablet_replica.cc:331] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:55.603765 7606 tablet_bootstrap.cc:492] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:55.604161 7606 ts_tablet_manager.cc:1397] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.003s
I20250110 20:57:55.603468 6990 catalog_manager.cc:5526] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:55.605614 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet d7f69b85ab3e4b2587d5461506533067 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:55.606384 7606 raft_consensus.cc:357] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.607008 7606 raft_consensus.cc:383] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.607276 7606 raft_consensus.cc:738] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.607792 7615 raft_consensus.cc:2238] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:55.607827 7606 consensus_queue.cc:260] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.611336 7616 ts_tablet_manager.cc:1918] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.607580 7602 raft_consensus.cc:357] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.611711 7616 log.cc:1198] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/89212badb30649dfb65e51c57760f787
I20250110 20:57:55.611781 7602 raft_consensus.cc:383] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.612077 7602 raft_consensus.cc:738] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.612334 7616 ts_tablet_manager.cc:1939] T 89212badb30649dfb65e51c57760f787 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:55.614096 6989 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:55.613862 7602 consensus_queue.cc:260] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.614264 7615 pending_rounds.cc:62] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:57:55.614652 7615 pending_rounds.cc:69] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878798021578752 op_type: NO_OP noop_request { }
I20250110 20:57:55.616545 7615 raft_consensus.cc:2883] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:55.616946 7615 raft_consensus.cc:2267] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:55.621181 6989 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 consensus state has no leader
I20250110 20:57:55.628881 7615 ts_tablet_manager.cc:1905] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.630007 7606 ts_tablet_manager.cc:1428] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20250110 20:57:55.630899 7617 ts_tablet_manager.cc:1918] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.631253 7617 log.cc:1198] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/5a955851581349debfc895eed2411635
I20250110 20:57:55.631862 7617 ts_tablet_manager.cc:1939] T 5a955851581349debfc895eed2411635 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:55.636096 7602 ts_tablet_manager.cc:1428] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.044s user 0.005s sys 0.001s
I20250110 20:57:55.636029 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 30aa8fd175de4c3aac137ac51f1ae51d (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.637127 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30aa8fd175de4c3aac137ac51f1ae51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.638545 7609 tablet_bootstrap.cc:654] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.638520 7646 raft_consensus.cc:491] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:55.638928 7646 raft_consensus.cc:513] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.640707 7646 leader_election.cc:290] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.641392 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89212badb30649dfb65e51c57760f787" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:55.642010 7208 raft_consensus.cc:2463] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:55.643167 7287 leader_election.cc:304] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:55.645330 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 30aa8fd175de4c3aac137ac51f1ae51d (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.646397 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30aa8fd175de4c3aac137ac51f1ae51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.647370 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 30aa8fd175de4c3aac137ac51f1ae51d (DEFAULT_TABLE table=default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.648365 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30aa8fd175de4c3aac137ac51f1ae51d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.650082 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 89212badb30649dfb65e51c57760f787 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:55.650305 7616 tablet_replica.cc:331] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
W20250110 20:57:55.651010 7288 leader_election.cc:343] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 89212badb30649dfb65e51c57760f787
I20250110 20:57:55.651034 6989 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:55.651970 7646 raft_consensus.cc:2798] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:55.652316 7646 raft_consensus.cc:491] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:55.652622 7646 raft_consensus.cc:3054] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.656739 7616 raft_consensus.cc:2238] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.657179 7616 raft_consensus.cc:2267] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.656984 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 7d49b406d2e3438594c4d837422eb707 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.657969 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d49b406d2e3438594c4d837422eb707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.660486 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:55.662375 7617 tablet_replica.cc:331] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:55.662423 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 7d49b406d2e3438594c4d837422eb707 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.663475 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d49b406d2e3438594c4d837422eb707. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.663784 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 7d49b406d2e3438594c4d837422eb707 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.664770 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d49b406d2e3438594c4d837422eb707. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:57:55.672124 6980 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 consensus state has no leader
I20250110 20:57:55.677242 7606 tablet_bootstrap.cc:492] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:55.678623 7646 raft_consensus.cc:513] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.683255 7602 tablet_bootstrap.cc:492] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:55.684604 7606 tablet_bootstrap.cc:654] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.686211 7617 raft_consensus.cc:2238] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.686645 7617 raft_consensus.cc:2267] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.688148 7617 ts_tablet_manager.cc:1905] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.696744 7616 ts_tablet_manager.cc:1905] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.698310 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89212badb30649dfb65e51c57760f787" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:55.698871 7208 raft_consensus.cc:3054] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.704759 7208 raft_consensus.cc:2463] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:55.705772 7287 leader_election.cc:304] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
W20250110 20:57:55.713657 7288 leader_election.cc:343] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 89212badb30649dfb65e51c57760f787
I20250110 20:57:55.714619 7646 leader_election.cc:290] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.716439 7646 raft_consensus.cc:2798] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:55.717856 7602 tablet_bootstrap.cc:654] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.718302 7646 raft_consensus.cc:695] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:55.720691 7646 consensus_queue.cc:237] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.722143 7609 tablet_bootstrap.cc:492] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:55.722523 7609 ts_tablet_manager.cc:1397] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.218s user 0.013s sys 0.005s
I20250110 20:57:55.724702 7609 raft_consensus.cc:357] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.725370 7609 raft_consensus.cc:383] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.726732 7609 raft_consensus.cc:738] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.727658 7609 consensus_queue.cc:260] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.729835 7615 ts_tablet_manager.cc:1918] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:55.730190 7615 log.cc:1198] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ddbadacad9ec42f98b2bd839aafcafa7
I20250110 20:57:55.730831 7615 ts_tablet_manager.cc:1939] T ddbadacad9ec42f98b2bd839aafcafa7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:55.733155 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet ddbadacad9ec42f98b2bd839aafcafa7 (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:55.737455 6989 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:55.741732 7602 tablet_bootstrap.cc:492] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:55.734849 7606 tablet_bootstrap.cc:492] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:55.742169 7602 ts_tablet_manager.cc:1397] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.003s
I20250110 20:57:55.742556 7606 ts_tablet_manager.cc:1397] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.002s
I20250110 20:57:55.744915 7606 raft_consensus.cc:357] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.745596 7606 raft_consensus.cc:383] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.745962 7606 raft_consensus.cc:738] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.746641 7606 consensus_queue.cc:260] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.754293 7606 ts_tablet_manager.cc:1428] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250110 20:57:55.755146 7606 tablet_bootstrap.cc:492] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:55.758463 7615 tablet_replica.cc:331] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:55.759131 7615 raft_consensus.cc:2238] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.759532 7615 raft_consensus.cc:2267] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.742096 7616 ts_tablet_manager.cc:1918] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.742872 7609 ts_tablet_manager.cc:1428] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.020s user 0.006s sys 0.000s
I20250110 20:57:55.744359 7602 raft_consensus.cc:357] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.766150 7602 raft_consensus.cc:383] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.766404 7602 raft_consensus.cc:738] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.765614 7616 log.cc:1198] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ddbadacad9ec42f98b2bd839aafcafa7
I20250110 20:57:55.767946 7616 ts_tablet_manager.cc:1939] T ddbadacad9ec42f98b2bd839aafcafa7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:55.767087 7617 ts_tablet_manager.cc:1918] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.777011 7615 ts_tablet_manager.cc:1905] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.777312 7617 log.cc:1198] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/d7f69b85ab3e4b2587d5461506533067
I20250110 20:57:55.777680 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 118976382cf440eebb9d00973928491f from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:55.778102 7617 ts_tablet_manager.cc:1939] T d7f69b85ab3e4b2587d5461506533067 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:55.766978 7602 consensus_queue.cc:260] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.783785 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet d7f69b85ab3e4b2587d5461506533067 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:55.784521 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet ddbadacad9ec42f98b2bd839aafcafa7 (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:55.785287 7609 tablet_bootstrap.cc:492] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:55.787137 7606 tablet_bootstrap.cc:654] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.787077 6990 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:55.791433 7602 ts_tablet_manager.cc:1428] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.049s user 0.005s sys 0.001s
I20250110 20:57:55.792251 7602 tablet_bootstrap.cc:492] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:55.795790 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:55.797231 7617 tablet_replica.cc:331] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:55.798521 7609 tablet_bootstrap.cc:654] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.803315 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:55.808009 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:55.825614 7602 tablet_bootstrap.cc:654] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.826340 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 118976382cf440eebb9d00973928491f
I20250110 20:57:55.828025 6989 catalog_manager.cc:6596] ReplaceTablet: tablet 118976382cf440eebb9d00973928491f deleted and replaced by tablet b3658617fe79493784078cf80b0346e8
I20250110 20:57:55.835568 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 118976382cf440eebb9d00973928491f with delete_type TABLET_DATA_DELETED (replaced by tablet b3658617fe79493784078cf80b0346e8) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:55.837031 7609 tablet_bootstrap.cc:492] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:55.837065 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 118976382cf440eebb9d00973928491f with delete_type TABLET_DATA_DELETED (replaced by tablet b3658617fe79493784078cf80b0346e8) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:55.837510 7609 ts_tablet_manager.cc:1397] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.004s
I20250110 20:57:55.837831 7616 tablet_replica.cc:331] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:55.838459 7616 raft_consensus.cc:2238] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:55.839123 7616 pending_rounds.cc:62] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Trying to abort 1 pending ops.
I20250110 20:57:55.839325 7616 pending_rounds.cc:69] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878798848176128 op_type: NO_OP noop_request { }
I20250110 20:57:55.839646 7616 raft_consensus.cc:2883] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:55.839983 7616 raft_consensus.cc:2267] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.839778 7609 raft_consensus.cc:357] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.840596 7609 raft_consensus.cc:383] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.840849 7609 raft_consensus.cc:738] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.841473 7609 consensus_queue.cc:260] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.842535 7616 ts_tablet_manager.cc:1905] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.845088 7602 tablet_bootstrap.cc:492] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:55.845479 7602 ts_tablet_manager.cc:1397] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.001s
I20250110 20:57:55.846876 7617 raft_consensus.cc:2238] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:55.847596 7602 raft_consensus.cc:357] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.848225 7602 raft_consensus.cc:383] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.848460 7602 raft_consensus.cc:738] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.850929 7609 ts_tablet_manager.cc:1428] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250110 20:57:55.851696 7609 tablet_bootstrap.cc:492] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:55.851686 7617 pending_rounds.cc:62] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:57:55.851939 7617 pending_rounds.cc:69] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878799867084800 op_type: NO_OP noop_request { }
I20250110 20:57:55.852278 7617 raft_consensus.cc:2883] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:55.852581 7617 raft_consensus.cc:2267] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.867285 7609 tablet_bootstrap.cc:654] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.871119 7615 ts_tablet_manager.cc:1918] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:55.871500 7615 log.cc:1198] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/89212badb30649dfb65e51c57760f787
I20250110 20:57:55.872098 7615 ts_tablet_manager.cc:1939] T 89212badb30649dfb65e51c57760f787 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:55.874492 7616 ts_tablet_manager.cc:1918] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:55.881841 7616 log.cc:1198] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/118976382cf440eebb9d00973928491f
I20250110 20:57:55.882396 7609 tablet_bootstrap.cc:492] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:55.882766 7609 ts_tablet_manager.cc:1397] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.001s
I20250110 20:57:55.883529 7616 ts_tablet_manager.cc:1939] T 118976382cf440eebb9d00973928491f P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:55.884817 7609 raft_consensus.cc:357] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.885432 7609 raft_consensus.cc:383] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.885680 7609 raft_consensus.cc:738] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.886648 7609 consensus_queue.cc:260] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.892690 7332 tablet_service.cc:1467] Processing CreateTablet for tablet b3658617fe79493784078cf80b0346e8 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.893870 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3658617fe79493784078cf80b0346e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.903002 7617 ts_tablet_manager.cc:1905] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.909943 7609 ts_tablet_manager.cc:1428] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20250110 20:57:55.911208 7609 tablet_bootstrap.cc:492] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:55.919714 7646 raft_consensus.cc:491] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:55.920394 7646 raft_consensus.cc:513] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.922138 7609 tablet_bootstrap.cc:654] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.923489 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a632c3126b01431abf5772f192753040" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:55.924129 7501 raft_consensus.cc:2463] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:55.925647 7288 leader_election.cc:304] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:55.926467 7646 leader_election.cc:290] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:55.926165 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 118976382cf440eebb9d00973928491f with delete_type TABLET_DATA_DELETED (replaced by tablet b3658617fe79493784078cf80b0346e8) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:55.926887 7646 raft_consensus.cc:2798] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:55.927156 7646 raft_consensus.cc:491] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:55.926683 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a632c3126b01431abf5772f192753040" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:55.927433 7646 raft_consensus.cc:3054] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.928225 7190 tablet_service.cc:1467] Processing CreateTablet for tablet b3658617fe79493784078cf80b0346e8 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.848989 7602 consensus_queue.cc:260] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.930446 7208 raft_consensus.cc:2463] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:55.930755 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3658617fe79493784078cf80b0346e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.939523 7477 tablet_service.cc:1467] Processing CreateTablet for tablet b3658617fe79493784078cf80b0346e8 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:55.940591 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3658617fe79493784078cf80b0346e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:55.942627 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 118976382cf440eebb9d00973928491f (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:55.935536 7602 ts_tablet_manager.cc:1428] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.090s user 0.001s sys 0.005s
I20250110 20:57:55.953367 7602 tablet_bootstrap.cc:492] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:55.955238 7606 tablet_bootstrap.cc:492] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:55.955691 7606 ts_tablet_manager.cc:1397] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.201s user 0.019s sys 0.019s
I20250110 20:57:55.957940 7606 raft_consensus.cc:357] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.961824 7606 raft_consensus.cc:383] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.961966 7609 tablet_bootstrap.cc:492] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:55.962342 7609 ts_tablet_manager.cc:1397] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.051s user 0.008s sys 0.000s
I20250110 20:57:55.964421 7609 raft_consensus.cc:357] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.965054 7609 raft_consensus.cc:383] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:55.965284 7609 raft_consensus.cc:738] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.966194 7609 consensus_queue.cc:260] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.963994 7606 raft_consensus.cc:738] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:55.967449 7606 consensus_queue.cc:260] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.972769 7615 tablet_replica.cc:331] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:55.974486 7615 raft_consensus.cc:2238] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:55.974510 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 89212badb30649dfb65e51c57760f787 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:55.974960 7615 raft_consensus.cc:2267] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:55.975330 7646 raft_consensus.cc:513] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:55.976922 7615 ts_tablet_manager.cc:1905] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:55.980297 7606 ts_tablet_manager.cc:1428] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20250110 20:57:55.981148 7606 tablet_bootstrap.cc:492] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:55.982149 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a632c3126b01431abf5772f192753040" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:55.982674 7208 raft_consensus.cc:3054] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:55.989423 7602 tablet_bootstrap.cc:654] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:55.991844 7208 raft_consensus.cc:2463] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:55.993093 7287 leader_election.cc:304] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:55.994041 7718 raft_consensus.cc:2798] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:56.003594 7609 ts_tablet_manager.cc:1428] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.041s user 0.006s sys 0.000s
I20250110 20:57:56.009711 7606 tablet_bootstrap.cc:654] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.013358 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:56.019829 7609 tablet_bootstrap.cc:492] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:55.970444 7617 ts_tablet_manager.cc:1918] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:56.023655 7617 log.cc:1198] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/89212badb30649dfb65e51c57760f787
I20250110 20:57:56.031231 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a632c3126b01431abf5772f192753040" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:56.031783 7501 raft_consensus.cc:3054] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.033334 7609 tablet_bootstrap.cc:654] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.040400 7602 tablet_bootstrap.cc:492] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:56.040786 7602 ts_tablet_manager.cc:1397] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.088s user 0.009s sys 0.000s
I20250110 20:57:56.044049 7615 ts_tablet_manager.cc:1918] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.043928 7602 raft_consensus.cc:357] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.044437 7615 log.cc:1198] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/118976382cf440eebb9d00973928491f
I20250110 20:57:56.044637 7602 raft_consensus.cc:383] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.044914 7602 raft_consensus.cc:738] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.045194 7615 ts_tablet_manager.cc:1939] T 118976382cf440eebb9d00973928491f P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:56.045675 7602 consensus_queue.cc:260] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.047376 7501 raft_consensus.cc:2463] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:56.049124 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet a632c3126b01431abf5772f192753040 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:56.049846 7617 ts_tablet_manager.cc:1939] T 89212badb30649dfb65e51c57760f787 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.053431 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 89212badb30649dfb65e51c57760f787 (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:56.050434 7646 leader_election.cc:290] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.057621 7718 raft_consensus.cc:695] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:56.058796 7718 consensus_queue.cc:237] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.059643 7602 ts_tablet_manager.cc:1428] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.019s user 0.003s sys 0.003s
I20250110 20:57:56.063445 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 118976382cf440eebb9d00973928491f (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:56.064771 7609 tablet_bootstrap.cc:492] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:56.065183 7609 ts_tablet_manager.cc:1397] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.003s
I20250110 20:57:56.067391 7609 raft_consensus.cc:357] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.068024 7609 raft_consensus.cc:383] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.068280 7609 raft_consensus.cc:738] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.073303 7617 tablet_replica.cc:331] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:56.079166 7602 tablet_bootstrap.cc:492] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:56.068836 7609 consensus_queue.cc:260] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.088768 7609 ts_tablet_manager.cc:1428] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20250110 20:57:56.112679 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet a632c3126b01431abf5772f192753040
I20250110 20:57:56.114431 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet a632c3126b01431abf5772f192753040 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ff193d49094dc6bf3052d302e2bcd0) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:56.114837 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet a632c3126b01431abf5772f192753040 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ff193d49094dc6bf3052d302e2bcd0) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:56.115108 7615 tablet_replica.cc:331] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:56.115720 7615 raft_consensus.cc:2238] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.116127 7615 raft_consensus.cc:2267] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.117684 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet a632c3126b01431abf5772f192753040 with delete_type TABLET_DATA_DELETED (replaced by tablet f2ff193d49094dc6bf3052d302e2bcd0) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:56.118345 7616 tablet_replica.cc:331] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:56.118968 7616 raft_consensus.cc:2238] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.119359 7616 raft_consensus.cc:2267] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.121845 7616 ts_tablet_manager.cc:1905] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.122035 7602 tablet_bootstrap.cc:654] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.122733 7617 raft_consensus.cc:2238] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.123173 7617 raft_consensus.cc:2267] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.126426 7615 ts_tablet_manager.cc:1905] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.129019 7617 ts_tablet_manager.cc:1905] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.133615 7718 raft_consensus.cc:491] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:56.134045 7718 raft_consensus.cc:513] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.136611 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aa8fd175de4c3aac137ac51f1ae51d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:56.137214 7208 raft_consensus.cc:2463] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:56.138288 7287 leader_election.cc:304] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:56.139286 7718 leader_election.cc:290] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.139700 7718 raft_consensus.cc:2798] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:56.139995 7718 raft_consensus.cc:491] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:56.140287 7718 raft_consensus.cc:3054] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.141656 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aa8fd175de4c3aac137ac51f1ae51d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:56.142249 7501 raft_consensus.cc:2463] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:56.143265 7606 tablet_bootstrap.cc:492] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:56.143626 7606 ts_tablet_manager.cc:1397] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.163s user 0.012s sys 0.014s
I20250110 20:57:56.145799 7606 raft_consensus.cc:357] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.146430 7606 raft_consensus.cc:383] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.146670 7606 raft_consensus.cc:738] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.150581 6989 catalog_manager.cc:6596] ReplaceTablet: tablet a632c3126b01431abf5772f192753040 deleted and replaced by tablet f2ff193d49094dc6bf3052d302e2bcd0
I20250110 20:57:56.151342 7606 consensus_queue.cc:260] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.159477 7602 tablet_bootstrap.cc:492] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:56.159861 7602 ts_tablet_manager.cc:1397] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.003s
I20250110 20:57:56.162149 7602 raft_consensus.cc:357] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.162739 7602 raft_consensus.cc:383] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.162976 7602 raft_consensus.cc:738] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.163559 7602 consensus_queue.cc:260] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.173059 7602 ts_tablet_manager.cc:1428] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.013s user 0.005s sys 0.001s
I20250110 20:57:56.176841 7602 tablet_bootstrap.cc:492] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:56.178958 7616 ts_tablet_manager.cc:1918] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.179323 7616 log.cc:1198] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/a632c3126b01431abf5772f192753040
I20250110 20:57:56.179888 7616 ts_tablet_manager.cc:1939] T a632c3126b01431abf5772f192753040 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:56.185992 7615 ts_tablet_manager.cc:1918] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.185722 7718 raft_consensus.cc:513] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.186345 7615 log.cc:1198] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/a632c3126b01431abf5772f192753040
I20250110 20:57:56.188238 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aa8fd175de4c3aac137ac51f1ae51d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:56.188581 7615 ts_tablet_manager.cc:1939] T a632c3126b01431abf5772f192753040 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:56.188788 7208 raft_consensus.cc:3054] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.190933 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet a632c3126b01431abf5772f192753040 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:56.194271 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet a632c3126b01431abf5772f192753040 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:56.195875 7602 tablet_bootstrap.cc:654] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.196014 7208 raft_consensus.cc:2463] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:56.197019 7287 leader_election.cc:304] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:56.198129 7731 raft_consensus.cc:2798] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:56.198568 7731 raft_consensus.cc:695] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:56.199304 7731 consensus_queue.cc:237] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.202292 7190 tablet_service.cc:1467] Processing CreateTablet for tablet f2ff193d49094dc6bf3052d302e2bcd0 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.203194 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ff193d49094dc6bf3052d302e2bcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.207280 7606 ts_tablet_manager.cc:1428] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.063s user 0.006s sys 0.001s
I20250110 20:57:56.216670 7602 tablet_bootstrap.cc:492] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:56.216444 7332 tablet_service.cc:1467] Processing CreateTablet for tablet f2ff193d49094dc6bf3052d302e2bcd0 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.217063 7602 ts_tablet_manager.cc:1397] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.001s
I20250110 20:57:56.217520 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ff193d49094dc6bf3052d302e2bcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.224684 7718 leader_election.cc:290] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.228574 7718 raft_consensus.cc:491] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:56.228971 7718 raft_consensus.cc:513] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.229439 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30aa8fd175de4c3aac137ac51f1ae51d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:56.229996 7501 raft_consensus.cc:3054] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.231472 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb2fe4e8fa48d2b4a76a8d50f2c025" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:56.232084 7208 raft_consensus.cc:2463] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:56.233039 7287 leader_election.cc:304] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:56.234149 7646 raft_consensus.cc:2798] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:56.234472 7646 raft_consensus.cc:491] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:56.234792 7646 raft_consensus.cc:3054] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.238307 7501 raft_consensus.cc:2463] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:56.241158 7646 raft_consensus.cc:513] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.242015 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:56.221997 7602 raft_consensus.cc:357] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.242336 7602 raft_consensus.cc:383] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.242602 7602 raft_consensus.cc:738] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.243692 7646 leader_election.cc:290] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.244385 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb2fe4e8fa48d2b4a76a8d50f2c025" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:56.244899 7208 raft_consensus.cc:3054] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.244962 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb2fe4e8fa48d2b4a76a8d50f2c025" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:56.245525 7500 raft_consensus.cc:3054] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.251502 7477 tablet_service.cc:1467] Processing CreateTablet for tablet f2ff193d49094dc6bf3052d302e2bcd0 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.252554 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2ff193d49094dc6bf3052d302e2bcd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.260007 7208 raft_consensus.cc:2463] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:56.261034 7287 leader_election.cc:304] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:56.261938 7646 raft_consensus.cc:2798] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:56.263659 7500 raft_consensus.cc:2463] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:56.269711 7609 tablet_bootstrap.cc:492] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:56.244215 7602 consensus_queue.cc:260] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.270479 7718 leader_election.cc:290] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.271399 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3cb2fe4e8fa48d2b4a76a8d50f2c025" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:56.273289 7500 raft_consensus.cc:2371] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1. Re-sending same reply.
I20250110 20:57:56.274616 7617 ts_tablet_manager.cc:1918] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.274947 7617 log.cc:1198] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ddbadacad9ec42f98b2bd839aafcafa7
I20250110 20:57:56.275111 7609 tablet_bootstrap.cc:654] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.275527 7617 ts_tablet_manager.cc:1939] T ddbadacad9ec42f98b2bd839aafcafa7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.277607 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet ddbadacad9ec42f98b2bd839aafcafa7 (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:56.285993 7609 tablet_bootstrap.cc:492] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:56.286069 7602 ts_tablet_manager.cc:1428] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.069s user 0.001s sys 0.004s
I20250110 20:57:56.286374 7609 ts_tablet_manager.cc:1397] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20250110 20:57:56.288720 7609 raft_consensus.cc:357] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.289350 7609 raft_consensus.cc:383] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.289616 7609 raft_consensus.cc:738] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.294095 7609 consensus_queue.cc:260] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.298071 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025
I20250110 20:57:56.298373 7617 tablet_replica.cc:331] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:56.305533 7646 raft_consensus.cc:695] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:56.306358 7646 consensus_queue.cc:237] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.307359 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 with delete_type TABLET_DATA_DELETED (replaced by tablet 6633402d9ab042acb750d47ad2dd70ee) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:56.308032 7615 tablet_replica.cc:331] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:56.308290 7617 raft_consensus.cc:2238] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.308657 7615 raft_consensus.cc:2238] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.308729 7617 raft_consensus.cc:2267] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.309036 7615 raft_consensus.cc:2267] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.310649 7615 ts_tablet_manager.cc:1905] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.308789 7738 raft_consensus.cc:491] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:56.313741 6989 catalog_manager.cc:6596] ReplaceTablet: tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 deleted and replaced by tablet 6633402d9ab042acb750d47ad2dd70ee
I20250110 20:57:56.314652 7609 ts_tablet_manager.cc:1428] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250110 20:57:56.315196 7738 raft_consensus.cc:513] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.324501 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d49b406d2e3438594c4d837422eb707" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:56.325081 7208 raft_consensus.cc:2463] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:56.326089 7287 leader_election.cc:304] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:56.326361 7615 ts_tablet_manager.cc:1918] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.326699 7615 log.cc:1198] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/b3cb2fe4e8fa48d2b4a76a8d50f2c025
I20250110 20:57:56.326758 7646 raft_consensus.cc:2798] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:56.327279 7615 ts_tablet_manager.cc:1939] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:56.313645 7617 ts_tablet_manager.cc:1905] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.327054 7646 raft_consensus.cc:491] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:56.331013 7646 raft_consensus.cc:3054] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.336114 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 with delete_type TABLET_DATA_DELETED (replaced by tablet 6633402d9ab042acb750d47ad2dd70ee) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:56.338295 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:56.339792 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 with delete_type TABLET_DATA_DELETED (replaced by tablet 6633402d9ab042acb750d47ad2dd70ee) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:56.340484 7616 tablet_replica.cc:331] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:56.341113 7616 raft_consensus.cc:2238] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.341534 7616 raft_consensus.cc:2267] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.343436 7616 ts_tablet_manager.cc:1905] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.348755 7602 tablet_bootstrap.cc:492] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:56.349798 7738 leader_election.cc:290] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.351050 7606 tablet_bootstrap.cc:492] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:56.351486 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d49b406d2e3438594c4d837422eb707" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:56.352098 7500 raft_consensus.cc:2463] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:56.357437 7606 tablet_bootstrap.cc:654] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.357471 7697 raft_consensus.cc:491] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:56.357870 7697 raft_consensus.cc:513] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.359692 7697 leader_election.cc:290] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:56.360126 7602 tablet_bootstrap.cc:654] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.361996 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ff193d49094dc6bf3052d302e2bcd0" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:56.362386 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2ff193d49094dc6bf3052d302e2bcd0" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
W20250110 20:57:56.363348 7435 leader_election.cc:343] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:56.365247 7436 leader_election.cc:343] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:56.365646 7436 leader_election.cc:304] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:57:56.366555 7697 raft_consensus.cc:2743] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:56.370190 7646 raft_consensus.cc:513] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.374194 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d49b406d2e3438594c4d837422eb707" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:56.374740 7208 raft_consensus.cc:3054] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.378537 7602 tablet_bootstrap.cc:492] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:56.379390 7602 ts_tablet_manager.cc:1397] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.002s
I20250110 20:57:56.381341 7208 raft_consensus.cc:2463] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:56.388335 7287 leader_election.cc:304] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:56.390247 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d49b406d2e3438594c4d837422eb707" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:56.390825 7500 raft_consensus.cc:3054] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.392242 7611 raft_consensus.cc:491] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:56.392663 7611 raft_consensus.cc:513] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.393883 7738 raft_consensus.cc:2798] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:56.394493 7611 leader_election.cc:290] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.395030 7738 raft_consensus.cc:695] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:56.396263 7646 leader_election.cc:290] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.396325 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3658617fe79493784078cf80b0346e8" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:56.396979 7353 raft_consensus.cc:2463] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:56.397881 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3658617fe79493784078cf80b0346e8" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:56.398456 7501 raft_consensus.cc:2463] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:56.399533 7145 leader_election.cc:304] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:56.399716 7738 consensus_queue.cc:237] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.400964 7611 raft_consensus.cc:2798] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:56.401299 7611 raft_consensus.cc:491] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:56.401683 7611 raft_consensus.cc:3054] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.402462 6990 catalog_manager.cc:5526] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:56.404589 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:56.411875 7611 raft_consensus.cc:513] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.419082 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3658617fe79493784078cf80b0346e8" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:56.419651 7353 raft_consensus.cc:3054] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.422314 7611 leader_election.cc:290] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:56.422374 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3658617fe79493784078cf80b0346e8" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:56.422993 7501 raft_consensus.cc:3054] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:56.425323 7602 raft_consensus.cc:357] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.427414 7616 ts_tablet_manager.cc:1918] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.433557 7616 log.cc:1198] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/b3cb2fe4e8fa48d2b4a76a8d50f2c025
I20250110 20:57:56.434301 7616 ts_tablet_manager.cc:1939] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:56.434597 7602 raft_consensus.cc:383] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.436796 7602 raft_consensus.cc:738] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.433081 7500 raft_consensus.cc:2463] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:56.437465 7602 consensus_queue.cc:260] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.441253 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:56.441495 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 6633402d9ab042acb750d47ad2dd70ee (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.442428 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6633402d9ab042acb750d47ad2dd70ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.444314 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 6633402d9ab042acb750d47ad2dd70ee (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.445230 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6633402d9ab042acb750d47ad2dd70ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.446055 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 6633402d9ab042acb750d47ad2dd70ee (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.447000 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6633402d9ab042acb750d47ad2dd70ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.447038 7606 tablet_bootstrap.cc:492] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:56.447691 7606 ts_tablet_manager.cc:1397] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.097s user 0.006s sys 0.009s
I20250110 20:57:56.449847 7606 raft_consensus.cc:357] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.450474 7606 raft_consensus.cc:383] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.450718 7606 raft_consensus.cc:738] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.451288 7606 consensus_queue.cc:260] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.452208 7501 raft_consensus.cc:2463] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:56.454447 7353 raft_consensus.cc:2463] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:56.455190 7145 leader_election.cc:304] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:56.458613 7617 ts_tablet_manager.cc:1918] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.458956 7617 log.cc:1198] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/118976382cf440eebb9d00973928491f
I20250110 20:57:56.459543 7617 ts_tablet_manager.cc:1939] T 118976382cf440eebb9d00973928491f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.460443 7611 raft_consensus.cc:2798] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:56.463958 7611 raft_consensus.cc:695] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:57:56.464725 7611 consensus_queue.cc:237] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.466886 7606 ts_tablet_manager.cc:1428] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250110 20:57:56.473553 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 118976382cf440eebb9d00973928491f (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:56.481424 7617 tablet_replica.cc:331] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:56.482084 7617 raft_consensus.cc:2238] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:56.482697 7617 pending_rounds.cc:62] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:57:56.482900 7617 pending_rounds.cc:69] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878801147170816 op_type: NO_OP noop_request { }
I20250110 20:57:56.483246 7617 raft_consensus.cc:2883] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:56.483528 7617 raft_consensus.cc:2267] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.488416 7602 ts_tablet_manager.cc:1428] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.099s user 0.005s sys 0.000s
I20250110 20:57:56.493537 7609 tablet_bootstrap.cc:492] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:56.493381 6989 catalog_manager.cc:5526] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:56.498965 7609 tablet_bootstrap.cc:654] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.500758 7617 ts_tablet_manager.cc:1905] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.514394 7602 tablet_bootstrap.cc:492] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:56.520550 7602 tablet_bootstrap.cc:654] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.542482 7602 tablet_bootstrap.cc:492] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:56.542862 7602 ts_tablet_manager.cc:1397] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.003s
I20250110 20:57:56.544477 6990 catalog_manager.cc:5526] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:56.554538 7617 ts_tablet_manager.cc:1918] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:56.554886 7617 log.cc:1198] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/a632c3126b01431abf5772f192753040
I20250110 20:57:56.555536 7617 ts_tablet_manager.cc:1939] T a632c3126b01431abf5772f192753040 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.544900 7602 raft_consensus.cc:357] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.561340 7602 raft_consensus.cc:383] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.561643 7602 raft_consensus.cc:738] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.562490 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet a632c3126b01431abf5772f192753040 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:56.562253 7602 consensus_queue.cc:260] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.563297 7617 tablet_replica.cc:331] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:56.563876 7617 raft_consensus.cc:2238] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:56.564455 7617 pending_rounds.cc:62] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:57:56.564649 7617 pending_rounds.cc:69] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878802181758976 op_type: NO_OP noop_request { }
I20250110 20:57:56.564967 7617 raft_consensus.cc:2883] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:56.565275 7617 raft_consensus.cc:2267] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.568511 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 7d49b406d2e3438594c4d837422eb707 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:56.572212 7602 ts_tablet_manager.cc:1428] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250110 20:57:56.583335 7617 ts_tablet_manager.cc:1905] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.589377 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 7d49b406d2e3438594c4d837422eb707
I20250110 20:57:56.595919 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d49b406d2e3438594c4d837422eb707 with delete_type TABLET_DATA_DELETED (replaced by tablet d3a55ad10d794ff7bb7aecb26f782aeb) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:56.596352 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 7d49b406d2e3438594c4d837422eb707 deleted and replaced by tablet d3a55ad10d794ff7bb7aecb26f782aeb
I20250110 20:57:56.597187 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d49b406d2e3438594c4d837422eb707 with delete_type TABLET_DATA_DELETED (replaced by tablet d3a55ad10d794ff7bb7aecb26f782aeb) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:56.598152 7616 tablet_replica.cc:331] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:56.598824 7616 raft_consensus.cc:2238] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.599257 7616 raft_consensus.cc:2267] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.600699 7616 ts_tablet_manager.cc:1905] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.629855 7189 tablet_service.cc:1467] Processing CreateTablet for tablet d3a55ad10d794ff7bb7aecb26f782aeb (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.630992 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a55ad10d794ff7bb7aecb26f782aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.634245 7333 tablet_service.cc:1467] Processing CreateTablet for tablet d3a55ad10d794ff7bb7aecb26f782aeb (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.635253 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a55ad10d794ff7bb7aecb26f782aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.638711 7477 tablet_service.cc:1467] Processing CreateTablet for tablet d3a55ad10d794ff7bb7aecb26f782aeb (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.639055 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:56.639487 7609 tablet_bootstrap.cc:492] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:56.639719 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3a55ad10d794ff7bb7aecb26f782aeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.639933 7609 ts_tablet_manager.cc:1397] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.146s user 0.020s sys 0.015s
I20250110 20:57:56.640833 7606 tablet_bootstrap.cc:492] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:56.642241 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d49b406d2e3438594c4d837422eb707 with delete_type TABLET_DATA_DELETED (replaced by tablet d3a55ad10d794ff7bb7aecb26f782aeb) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:56.642241 7609 raft_consensus.cc:357] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.642872 7615 tablet_replica.cc:331] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:56.642870 7609 raft_consensus.cc:383] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.643128 7609 raft_consensus.cc:738] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.643541 7615 raft_consensus.cc:2238] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.643970 7615 raft_consensus.cc:2267] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.643669 7609 consensus_queue.cc:260] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.645851 7615 ts_tablet_manager.cc:1905] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.646142 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:56.648346 7606 tablet_bootstrap.cc:654] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.653945 7617 ts_tablet_manager.cc:1918] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:56.654290 7617 log.cc:1198] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/b3cb2fe4e8fa48d2b4a76a8d50f2c025
I20250110 20:57:56.654930 7617 ts_tablet_manager.cc:1939] T b3cb2fe4e8fa48d2b4a76a8d50f2c025 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.657143 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet b3cb2fe4e8fa48d2b4a76a8d50f2c025 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:56.661535 7606 tablet_bootstrap.cc:492] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:56.661895 7606 ts_tablet_manager.cc:1397] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.003s
I20250110 20:57:56.663960 7606 raft_consensus.cc:357] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.665198 7606 raft_consensus.cc:383] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.665473 7606 raft_consensus.cc:738] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.666003 7606 consensus_queue.cc:260] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.667834 7606 ts_tablet_manager.cc:1428] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20250110 20:57:56.679432 6990 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6633402d9ab042acb750d47ad2dd70ee consensus state has no leader
I20250110 20:57:56.681775 6989 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:56.682829 7609 ts_tablet_manager.cc:1428] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.043s user 0.008s sys 0.002s
I20250110 20:57:56.683017 7617 tablet_replica.cc:331] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:56.685249 7617 raft_consensus.cc:2238] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:56.693100 7616 ts_tablet_manager.cc:1918] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.693792 7616 log.cc:1198] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/7d49b406d2e3438594c4d837422eb707
I20250110 20:57:56.693751 7617 pending_rounds.cc:62] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:57:56.694648 7617 pending_rounds.cc:69] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878802591715328 op_type: NO_OP noop_request { }
I20250110 20:57:56.694773 6990 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:56.694998 7617 raft_consensus.cc:2883] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:56.695470 7616 ts_tablet_manager.cc:1939] T 7d49b406d2e3438594c4d837422eb707 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:56.695302 7617 raft_consensus.cc:2267] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.698602 7606 tablet_bootstrap.cc:492] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
W20250110 20:57:56.700801 6980 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6633402d9ab042acb750d47ad2dd70ee consensus state has no leader
I20250110 20:57:56.702348 7617 ts_tablet_manager.cc:1905] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:56.703653 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:56.703809 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 7d49b406d2e3438594c4d837422eb707 (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:56.704542 6990 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:56.704991 7609 tablet_bootstrap.cc:492] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:56.708879 6989 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:56.710327 7602 tablet_bootstrap.cc:492] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:56.712337 7615 ts_tablet_manager.cc:1918] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.713349 7615 log.cc:1198] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/7d49b406d2e3438594c4d837422eb707
I20250110 20:57:56.713452 7606 tablet_bootstrap.cc:654] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.714352 7615 ts_tablet_manager.cc:1939] T 7d49b406d2e3438594c4d837422eb707 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:56.715696 7609 tablet_bootstrap.cc:654] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.715217 7602 tablet_bootstrap.cc:654] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.727823 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 7d49b406d2e3438594c4d837422eb707 (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:56.730466 7602 tablet_bootstrap.cc:492] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:56.730835 7602 ts_tablet_manager.cc:1397] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.000s
I20250110 20:57:56.732913 7602 raft_consensus.cc:357] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.733566 7602 raft_consensus.cc:383] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.733803 7602 raft_consensus.cc:738] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.734468 7602 consensus_queue.cc:260] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.736948 7609 tablet_bootstrap.cc:492] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:56.737354 7609 ts_tablet_manager.cc:1397] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.000s
I20250110 20:57:56.747119 7602 ts_tablet_manager.cc:1428] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20250110 20:57:56.739451 7609 raft_consensus.cc:357] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.747840 7609 raft_consensus.cc:383] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.748098 7609 raft_consensus.cc:738] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.748646 7609 consensus_queue.cc:260] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.751278 7617 ts_tablet_manager.cc:1918] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:56.751646 7617 log.cc:1198] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/7d49b406d2e3438594c4d837422eb707
I20250110 20:57:56.752276 7617 ts_tablet_manager.cc:1939] T 7d49b406d2e3438594c4d837422eb707 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.756768 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 7d49b406d2e3438594c4d837422eb707 (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:56.759105 7609 ts_tablet_manager.cc:1428] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.021s user 0.000s sys 0.004s
I20250110 20:57:56.768326 7606 tablet_bootstrap.cc:492] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:56.768698 7606 ts_tablet_manager.cc:1397] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.070s user 0.023s sys 0.021s
I20250110 20:57:56.770859 7606 raft_consensus.cc:357] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.785820 7606 raft_consensus.cc:383] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:56.786098 7606 raft_consensus.cc:738] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:56.786731 7606 consensus_queue.cc:260] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:56.809453 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet f2ff193d49094dc6bf3052d302e2bcd0 from {username='slave'} at 127.0.0.1:33774
W20250110 20:57:56.821707 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:56.833220 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet f2ff193d49094dc6bf3052d302e2bcd0
I20250110 20:57:56.836580 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ff193d49094dc6bf3052d302e2bcd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 25b2023ef65a4cceadc07f05f1839309) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:56.836768 6989 catalog_manager.cc:6596] ReplaceTablet: tablet f2ff193d49094dc6bf3052d302e2bcd0 deleted and replaced by tablet 25b2023ef65a4cceadc07f05f1839309
I20250110 20:57:56.837298 7615 tablet_replica.cc:331] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:56.837972 7615 raft_consensus.cc:2238] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.838363 7615 raft_consensus.cc:2267] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.839498 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ff193d49094dc6bf3052d302e2bcd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 25b2023ef65a4cceadc07f05f1839309) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:56.840142 7616 tablet_replica.cc:331] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:56.840778 7616 raft_consensus.cc:2238] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.841181 7616 raft_consensus.cc:2267] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.842697 7616 ts_tablet_manager.cc:1905] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.796998 7606 ts_tablet_manager.cc:1428] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20250110 20:57:56.846771 7615 ts_tablet_manager.cc:1905] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.839010 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet f2ff193d49094dc6bf3052d302e2bcd0 with delete_type TABLET_DATA_DELETED (replaced by tablet 25b2023ef65a4cceadc07f05f1839309) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:56.855427 7617 tablet_replica.cc:331] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:56.856071 7617 raft_consensus.cc:2238] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.856488 7617 raft_consensus.cc:2267] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.858412 7617 ts_tablet_manager.cc:1905] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.874657 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 6633402d9ab042acb750d47ad2dd70ee from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:56.890930 7615 ts_tablet_manager.cc:1918] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.894778 7617 ts_tablet_manager.cc:1918] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.895117 7617 log.cc:1198] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f2ff193d49094dc6bf3052d302e2bcd0
I20250110 20:57:56.895730 7617 ts_tablet_manager.cc:1939] T f2ff193d49094dc6bf3052d302e2bcd0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.897603 7615 log.cc:1198] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/f2ff193d49094dc6bf3052d302e2bcd0
I20250110 20:57:56.895709 7616 ts_tablet_manager.cc:1918] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.898186 7616 log.cc:1198] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/f2ff193d49094dc6bf3052d302e2bcd0
I20250110 20:57:56.899092 7615 ts_tablet_manager.cc:1939] T f2ff193d49094dc6bf3052d302e2bcd0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:56.900502 7616 ts_tablet_manager.cc:1939] T f2ff193d49094dc6bf3052d302e2bcd0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:56.904435 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet f2ff193d49094dc6bf3052d302e2bcd0 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:56.914259 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 6633402d9ab042acb750d47ad2dd70ee
I20250110 20:57:56.915535 6989 catalog_manager.cc:6596] ReplaceTablet: tablet 6633402d9ab042acb750d47ad2dd70ee deleted and replaced by tablet 534f0512726c4fa2882aa23332ad8ee7
I20250110 20:57:56.916222 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 6633402d9ab042acb750d47ad2dd70ee with delete_type TABLET_DATA_DELETED (replaced by tablet 534f0512726c4fa2882aa23332ad8ee7) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:56.919341 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 25b2023ef65a4cceadc07f05f1839309 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.920544 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25b2023ef65a4cceadc07f05f1839309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.927049 7615 tablet_replica.cc:331] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:56.927760 7615 raft_consensus.cc:2238] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.928304 7615 raft_consensus.cc:2267] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.930922 7615 ts_tablet_manager.cc:1905] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.941628 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet f2ff193d49094dc6bf3052d302e2bcd0 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:56.943010 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 6633402d9ab042acb750d47ad2dd70ee with delete_type TABLET_DATA_DELETED (replaced by tablet 534f0512726c4fa2882aa23332ad8ee7) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:56.943739 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 6633402d9ab042acb750d47ad2dd70ee with delete_type TABLET_DATA_DELETED (replaced by tablet 534f0512726c4fa2882aa23332ad8ee7) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:56.944438 7616 tablet_replica.cc:331] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:56.945087 7616 raft_consensus.cc:2238] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.946122 7617 tablet_replica.cc:331] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:56.946084 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 25b2023ef65a4cceadc07f05f1839309 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.946205 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 25b2023ef65a4cceadc07f05f1839309 (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:56.946808 7617 raft_consensus.cc:2238] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:56.947046 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25b2023ef65a4cceadc07f05f1839309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.947219 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25b2023ef65a4cceadc07f05f1839309. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:56.947219 7617 raft_consensus.cc:2267] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.949383 7617 ts_tablet_manager.cc:1905] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.949528 7616 raft_consensus.cc:2267] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:56.953749 7616 ts_tablet_manager.cc:1905] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:56.966230 7602 tablet_bootstrap.cc:492] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:56.971138 7602 tablet_bootstrap.cc:654] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.972659 7609 tablet_bootstrap.cc:492] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:56.975005 7617 ts_tablet_manager.cc:1918] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.975163 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet f2ff193d49094dc6bf3052d302e2bcd0 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:56.977555 7609 tablet_bootstrap.cc:654] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.980820 7617 log.cc:1198] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/6633402d9ab042acb750d47ad2dd70ee
I20250110 20:57:56.986186 7616 ts_tablet_manager.cc:1918] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.986526 7616 log.cc:1198] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/6633402d9ab042acb750d47ad2dd70ee
I20250110 20:57:56.986966 7617 ts_tablet_manager.cc:1939] T 6633402d9ab042acb750d47ad2dd70ee P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:56.988641 7615 ts_tablet_manager.cc:1918] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:56.982528 7606 tablet_bootstrap.cc:492] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:56.990917 7615 log.cc:1198] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/6633402d9ab042acb750d47ad2dd70ee
I20250110 20:57:56.991895 7615 ts_tablet_manager.cc:1939] T 6633402d9ab042acb750d47ad2dd70ee P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:56.994254 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:56.995355 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:56.997493 7616 ts_tablet_manager.cc:1939] T 6633402d9ab042acb750d47ad2dd70ee P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:56.997869 7606 tablet_bootstrap.cc:654] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:56.999975 7602 tablet_bootstrap.cc:492] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:57.000377 7602 ts_tablet_manager.cc:1397] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.000s
I20250110 20:57:57.010380 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:57.014696 7609 tablet_bootstrap.cc:492] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:57.015097 7609 ts_tablet_manager.cc:1397] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.007s
I20250110 20:57:57.015631 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.016640 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 534f0512726c4fa2882aa23332ad8ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.016909 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.017308 7609 raft_consensus.cc:357] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.017843 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 534f0512726c4fa2882aa23332ad8ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.017942 7609 raft_consensus.cc:383] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.018178 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.002508 7602 raft_consensus.cc:357] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.018642 7602 raft_consensus.cc:383] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.018882 7602 raft_consensus.cc:738] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.019069 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 534f0512726c4fa2882aa23332ad8ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.019484 7602 consensus_queue.cc:260] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.018178 7609 raft_consensus.cc:738] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.025592 7609 consensus_queue.cc:260] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.027798 7609 ts_tablet_manager.cc:1428] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.012s user 0.002s sys 0.003s
I20250110 20:57:57.033398 7738 consensus_queue.cc:1035] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:57.050770 7606 tablet_bootstrap.cc:492] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:57.051414 7606 ts_tablet_manager.cc:1397] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.003s
I20250110 20:57:57.054065 7602 ts_tablet_manager.cc:1428] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.053s user 0.006s sys 0.001s
I20250110 20:57:57.056658 7602 tablet_bootstrap.cc:492] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:57.068356 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 534f0512726c4fa2882aa23332ad8ee7 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:57.053936 7606 raft_consensus.cc:357] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.079630 7606 raft_consensus.cc:383] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.079926 7606 raft_consensus.cc:738] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.080540 7606 consensus_queue.cc:260] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.087719 7609 tablet_bootstrap.cc:492] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:57.101460 7606 ts_tablet_manager.cc:1428] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.050s user 0.008s sys 0.000s
I20250110 20:57:57.102356 7606 tablet_bootstrap.cc:492] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:57.106604 7609 tablet_bootstrap.cc:654] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.113700 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 534f0512726c4fa2882aa23332ad8ee7
I20250110 20:57:57.115005 6989 catalog_manager.cc:6596] ReplaceTablet: tablet 534f0512726c4fa2882aa23332ad8ee7 deleted and replaced by tablet 669acec9184749f094421b99bea0588f
I20250110 20:57:57.116919 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 with delete_type TABLET_DATA_DELETED (replaced by tablet 669acec9184749f094421b99bea0588f) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:57.117885 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 with delete_type TABLET_DATA_DELETED (replaced by tablet 669acec9184749f094421b99bea0588f) from {username='slave'} at 127.0.0.1:43396
W20250110 20:57:57.120023 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 534f0512726c4fa2882aa23332ad8ee7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 534f0512726c4fa2882aa23332ad8ee7 already in progress: creating tablet
W20250110 20:57:57.120183 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet 534f0512726c4fa2882aa23332ad8ee7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 534f0512726c4fa2882aa23332ad8ee7 already in progress: creating tablet
I20250110 20:57:57.121795 7609 tablet_bootstrap.cc:492] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:57.122179 7609 ts_tablet_manager.cc:1397] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.002s
I20250110 20:57:57.122326 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 with delete_type TABLET_DATA_DELETED (replaced by tablet 669acec9184749f094421b99bea0588f) from {username='slave'} at 127.0.0.1:40394
W20250110 20:57:57.126734 6980 catalog_manager.cc:4852] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet 534f0512726c4fa2882aa23332ad8ee7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 534f0512726c4fa2882aa23332ad8ee7 already in progress: creating tablet
I20250110 20:57:57.132719 7762 consensus_queue.cc:1035] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:57.134148 7606 tablet_bootstrap.cc:654] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.124243 7609 raft_consensus.cc:357] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.142050 7602 tablet_bootstrap.cc:654] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.145501 7609 raft_consensus.cc:383] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.145781 7609 raft_consensus.cc:738] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.146432 7609 consensus_queue.cc:260] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.147367 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:57.148407 7609 ts_tablet_manager.cc:1428] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20250110 20:57:57.178395 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 669acec9184749f094421b99bea0588f (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.179507 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669acec9184749f094421b99bea0588f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.185078 7606 tablet_bootstrap.cc:492] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:57.185492 7606 ts_tablet_manager.cc:1397] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.083s user 0.012s sys 0.001s
I20250110 20:57:57.187285 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 669acec9184749f094421b99bea0588f (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.187748 7606 raft_consensus.cc:357] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.188185 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669acec9184749f094421b99bea0588f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.188284 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 669acec9184749f094421b99bea0588f (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.189302 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 669acec9184749f094421b99bea0588f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.195209 7602 tablet_bootstrap.cc:492] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:57.201603 7602 ts_tablet_manager.cc:1397] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.147s user 0.009s sys 0.003s
I20250110 20:57:57.205533 7606 raft_consensus.cc:383] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.205853 7606 raft_consensus.cc:738] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.207408 7606 consensus_queue.cc:260] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.209334 7606 ts_tablet_manager.cc:1428] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20250110 20:57:57.214635 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 with delete_type TABLET_DATA_DELETED (replaced by tablet 669acec9184749f094421b99bea0588f) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:57.215211 7617 tablet_replica.cc:331] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:57.215826 7617 raft_consensus.cc:2238] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.216235 7617 raft_consensus.cc:2267] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.216359 7609 tablet_bootstrap.cc:492] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:57.203750 7602 raft_consensus.cc:357] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.218010 7617 ts_tablet_manager.cc:1905] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.221488 7602 raft_consensus.cc:383] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.222084 7602 raft_consensus.cc:738] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.225580 7609 tablet_bootstrap.cc:654] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.232630 7617 ts_tablet_manager.cc:1918] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:57.232971 7617 log.cc:1198] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/534f0512726c4fa2882aa23332ad8ee7
I20250110 20:57:57.242388 7606 tablet_bootstrap.cc:492] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:57.242157 7602 consensus_queue.cc:260] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:57:57.248344 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:57:57.248692 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:57.253158 7617 ts_tablet_manager.cc:1939] T 534f0512726c4fa2882aa23332ad8ee7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:57.253104 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 with delete_type TABLET_DATA_DELETED (replaced by tablet 669acec9184749f094421b99bea0588f) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:57.254714 7616 tablet_replica.cc:331] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:57.255064 7611 raft_consensus.cc:491] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:57.255410 7616 raft_consensus.cc:2238] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.255357 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 534f0512726c4fa2882aa23332ad8ee7 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:57.252357 7606 tablet_bootstrap.cc:654] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.255796 7616 raft_consensus.cc:2267] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.255481 7611 raft_consensus.cc:513] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.258184 7616 ts_tablet_manager.cc:1905] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.259320 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 with delete_type TABLET_DATA_DELETED (replaced by tablet 669acec9184749f094421b99bea0588f) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:57.259554 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a55ad10d794ff7bb7aecb26f782aeb" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:57.257210 7611 leader_election.cc:290] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:57.260151 7353 raft_consensus.cc:2463] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
W20250110 20:57:57.260553 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet 534f0512726c4fa2882aa23332ad8ee7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 534f0512726c4fa2882aa23332ad8ee7 already in progress: creating tablet
I20250110 20:57:57.261003 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a55ad10d794ff7bb7aecb26f782aeb" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:57.261596 7501 raft_consensus.cc:2463] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:57.262646 7145 leader_election.cc:304] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:57.263275 7611 raft_consensus.cc:2798] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:57.263589 7611 raft_consensus.cc:491] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:57.263957 7611 raft_consensus.cc:3054] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:57.270840 7606 tablet_bootstrap.cc:492] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:57.271226 7606 ts_tablet_manager.cc:1397] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.004s
I20250110 20:57:57.270778 7611 raft_consensus.cc:513] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.273357 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a55ad10d794ff7bb7aecb26f782aeb" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:57.273921 7353 raft_consensus.cc:3054] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:57.274135 7616 ts_tablet_manager.cc:1918] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:57.274461 7616 log.cc:1198] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/534f0512726c4fa2882aa23332ad8ee7
I20250110 20:57:57.275017 7616 ts_tablet_manager.cc:1939] T 534f0512726c4fa2882aa23332ad8ee7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:57.277113 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 534f0512726c4fa2882aa23332ad8ee7 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:57.280261 7353 raft_consensus.cc:2463] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:57.281057 7145 leader_election.cc:304] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:57.282618 7609 tablet_bootstrap.cc:492] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:57.283074 7609 ts_tablet_manager.cc:1397] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.010s
I20250110 20:57:57.287250 7609 raft_consensus.cc:357] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.287969 7609 raft_consensus.cc:383] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.288229 7609 raft_consensus.cc:738] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.288812 7609 consensus_queue.cc:260] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.294905 7772 raft_consensus.cc:2798] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:57.296030 7611 leader_election.cc:290] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:57.296144 7609 ts_tablet_manager.cc:1428] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20250110 20:57:57.296911 6989 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:57.297262 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3a55ad10d794ff7bb7aecb26f782aeb" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:57.297777 7501 raft_consensus.cc:3054] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:57.298861 6989 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 669acec9184749f094421b99bea0588f consensus state has no leader
I20250110 20:57:57.304649 7501 raft_consensus.cc:2463] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:57.305117 6989 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:57.307178 7602 ts_tablet_manager.cc:1428] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.105s user 0.005s sys 0.000s
I20250110 20:57:57.310241 7606 raft_consensus.cc:357] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.310920 7606 raft_consensus.cc:383] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.311156 7606 raft_consensus.cc:738] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.314360 7606 consensus_queue.cc:260] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.315789 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 534f0512726c4fa2882aa23332ad8ee7 with delete_type TABLET_DATA_DELETED (replaced by tablet 669acec9184749f094421b99bea0588f) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:57.316442 7615 tablet_replica.cc:331] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:57.317085 7615 raft_consensus.cc:2238] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.317490 7615 raft_consensus.cc:2267] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.318706 7615 ts_tablet_manager.cc:1905] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.324668 7606 ts_tablet_manager.cc:1428] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20250110 20:57:57.328708 6989 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:57.338444 6989 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:57.340742 7615 ts_tablet_manager.cc:1918] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:57.341081 7615 log.cc:1198] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/534f0512726c4fa2882aa23332ad8ee7
I20250110 20:57:57.341759 7773 consensus_queue.cc:1035] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:57:57.358569 7772 raft_consensus.cc:695] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:57:57.359781 7772 consensus_queue.cc:237] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.363212 7602 tablet_bootstrap.cc:492] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:57.374228 7611 consensus_queue.cc:1035] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:57.392232 7773 raft_consensus.cc:491] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:57.392731 7773 raft_consensus.cc:513] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.394659 7773 leader_election.cc:290] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:57.396347 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b2023ef65a4cceadc07f05f1839309" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:57.396729 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b2023ef65a4cceadc07f05f1839309" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:57.397058 7501 raft_consensus.cc:2463] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:57.397389 7352 raft_consensus.cc:2463] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:57.398408 7145 leader_election.cc:304] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:57.400839 7774 raft_consensus.cc:2798] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:57.401217 7774 raft_consensus.cc:491] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:57.401803 7774 raft_consensus.cc:3054] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:57.423517 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet b3658617fe79493784078cf80b0346e8 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:57.425792 7615 ts_tablet_manager.cc:1939] T 534f0512726c4fa2882aa23332ad8ee7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:57.430830 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 534f0512726c4fa2882aa23332ad8ee7 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:57.437309 7602 tablet_bootstrap.cc:654] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.439831 7774 raft_consensus.cc:513] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.443838 7774 leader_election.cc:290] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:57.444579 6989 catalog_manager.cc:5526] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:57.446795 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b2023ef65a4cceadc07f05f1839309" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:57.447223 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet b3658617fe79493784078cf80b0346e8
I20250110 20:57:57.447358 7501 raft_consensus.cc:3054] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:57.448469 6990 catalog_manager.cc:6596] ReplaceTablet: tablet b3658617fe79493784078cf80b0346e8 deleted and replaced by tablet 00f841d449b24c2ebe0c5cb708e4437d
I20250110 20:57:57.449609 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet b3658617fe79493784078cf80b0346e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 00f841d449b24c2ebe0c5cb708e4437d) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:57.450243 7617 tablet_replica.cc:331] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:57.450193 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet b3658617fe79493784078cf80b0346e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 00f841d449b24c2ebe0c5cb708e4437d) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:57.450845 7616 tablet_replica.cc:331] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:57.450945 7617 raft_consensus.cc:2238] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.451365 7617 raft_consensus.cc:2267] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.451493 7616 raft_consensus.cc:2238] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.451936 7616 raft_consensus.cc:2267] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.453620 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet b3658617fe79493784078cf80b0346e8 with delete_type TABLET_DATA_DELETED (replaced by tablet 00f841d449b24c2ebe0c5cb708e4437d) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:57.454254 7616 ts_tablet_manager.cc:1905] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.457520 7617 ts_tablet_manager.cc:1905] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.463941 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:57.466498 7501 raft_consensus.cc:2463] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:57.467756 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25b2023ef65a4cceadc07f05f1839309" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:57.468104 7145 leader_election.cc:304] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:57.468322 7353 raft_consensus.cc:3054] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:57.468873 7774 raft_consensus.cc:2798] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:57.469300 7774 raft_consensus.cc:695] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:57:57.470021 7774 consensus_queue.cc:237] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.473460 7616 ts_tablet_manager.cc:1918] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:57.473790 7616 log.cc:1198] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/b3658617fe79493784078cf80b0346e8
I20250110 20:57:57.474444 7616 ts_tablet_manager.cc:1939] T b3658617fe79493784078cf80b0346e8 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:57.477737 7615 tablet_replica.cc:331] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:57.478405 7615 raft_consensus.cc:2238] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:57.479187 7615 raft_consensus.cc:2267] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.480719 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet b3658617fe79493784078cf80b0346e8 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:57.487696 7615 ts_tablet_manager.cc:1905] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.492977 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 00f841d449b24c2ebe0c5cb708e4437d (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.493608 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 00f841d449b24c2ebe0c5cb708e4437d (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.493990 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f841d449b24c2ebe0c5cb708e4437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.494525 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f841d449b24c2ebe0c5cb708e4437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.494589 6989 catalog_manager.cc:5526] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:57.497747 7353 raft_consensus.cc:2463] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:57.502323 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 00f841d449b24c2ebe0c5cb708e4437d (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.503365 7602 tablet_bootstrap.cc:492] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:57.503371 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f841d449b24c2ebe0c5cb708e4437d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.503738 7602 ts_tablet_manager.cc:1397] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.000s
I20250110 20:57:57.508738 7617 ts_tablet_manager.cc:1918] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:57.509061 7617 log.cc:1198] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/b3658617fe79493784078cf80b0346e8
I20250110 20:57:57.509145 7606 tablet_bootstrap.cc:492] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:57.514614 7606 tablet_bootstrap.cc:654] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.525043 7602 raft_consensus.cc:357] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.525696 7602 raft_consensus.cc:383] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.525941 7602 raft_consensus.cc:738] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.527189 7606 tablet_bootstrap.cc:492] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:57.527261 7617 ts_tablet_manager.cc:1939] T b3658617fe79493784078cf80b0346e8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:57.527559 7606 ts_tablet_manager.cc:1397] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.002s
I20250110 20:57:57.529556 7606 raft_consensus.cc:357] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.530165 7606 raft_consensus.cc:383] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.530395 7606 raft_consensus.cc:738] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.526489 7602 consensus_queue.cc:260] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.535326 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet b3658617fe79493784078cf80b0346e8 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:57.536527 7602 ts_tablet_manager.cc:1428] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20250110 20:57:57.536207 7606 consensus_queue.cc:260] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.541173 6989 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:57.546867 6989 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:57.550237 7609 tablet_bootstrap.cc:492] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:57.553764 7602 tablet_bootstrap.cc:492] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:57.556483 7609 tablet_bootstrap.cc:654] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.559542 7602 tablet_bootstrap.cc:654] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.567981 7606 ts_tablet_manager.cc:1428] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.040s user 0.005s sys 0.000s
I20250110 20:57:57.576926 7609 tablet_bootstrap.cc:492] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:57.577293 7609 ts_tablet_manager.cc:1397] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.011s
I20250110 20:57:57.579313 7609 raft_consensus.cc:357] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.579933 7609 raft_consensus.cc:383] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.580178 7609 raft_consensus.cc:738] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.580739 7609 consensus_queue.cc:260] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.581954 7602 tablet_bootstrap.cc:492] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:57.582332 7602 ts_tablet_manager.cc:1397] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20250110 20:57:57.584568 7602 raft_consensus.cc:357] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.585181 7602 raft_consensus.cc:383] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.585449 7602 raft_consensus.cc:738] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.586861 7602 consensus_queue.cc:260] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.595091 7609 ts_tablet_manager.cc:1428] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.018s user 0.001s sys 0.003s
I20250110 20:57:57.589327 7615 ts_tablet_manager.cc:1918] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:57.597376 7615 log.cc:1198] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/b3658617fe79493784078cf80b0346e8
I20250110 20:57:57.598070 7615 ts_tablet_manager.cc:1939] T b3658617fe79493784078cf80b0346e8 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:57.608613 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet b3658617fe79493784078cf80b0346e8 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:57.589202 7602 ts_tablet_manager.cc:1428] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:57:57.658257 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet d3a55ad10d794ff7bb7aecb26f782aeb from {username='slave'} at 127.0.0.1:33774
W20250110 20:57:57.663436 7047 hms_client.cc:305] Time spent drop HMS table: real 2.970s user 0.000s sys 0.000s
I20250110 20:57:57.664515 6992 catalog_manager.cc:2686] deleted HMS catalog entry for table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]
I20250110 20:57:57.674005 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet d3a55ad10d794ff7bb7aecb26f782aeb
I20250110 20:57:57.675487 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a55ad10d794ff7bb7aecb26f782aeb with delete_type TABLET_DATA_DELETED (replaced by tablet 143a38cabcea48e8926d9edcbe681def) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:57.676157 7615 tablet_replica.cc:331] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:57.676687 6989 catalog_manager.cc:6596] ReplaceTablet: tablet d3a55ad10d794ff7bb7aecb26f782aeb deleted and replaced by tablet 143a38cabcea48e8926d9edcbe681def
I20250110 20:57:57.676806 7615 raft_consensus.cc:2238] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:57.677273 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a55ad10d794ff7bb7aecb26f782aeb with delete_type TABLET_DATA_DELETED (replaced by tablet 143a38cabcea48e8926d9edcbe681def) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:57.677395 7615 pending_rounds.cc:62] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:57:57.677630 7615 pending_rounds.cc:69] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878806545547264 op_type: NO_OP noop_request { }
I20250110 20:57:57.677968 7616 tablet_replica.cc:331] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:57.677989 7615 raft_consensus.cc:2883] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:57.678270 7615 raft_consensus.cc:2267] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.678597 7616 raft_consensus.cc:2238] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.678900 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet d3a55ad10d794ff7bb7aecb26f782aeb with delete_type TABLET_DATA_DELETED (replaced by tablet 143a38cabcea48e8926d9edcbe681def) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:57.679068 7616 raft_consensus.cc:2267] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.679488 7617 tablet_replica.cc:331] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:57.680127 7617 raft_consensus.cc:2238] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.680516 7617 raft_consensus.cc:2267] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.680591 7615 ts_tablet_manager.cc:1905] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.680724 7616 ts_tablet_manager.cc:1905] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.681990 7617 ts_tablet_manager.cc:1905] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.689244 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:57.696725 7615 ts_tablet_manager.cc:1918] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:57.697530 7615 log.cc:1198] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/d3a55ad10d794ff7bb7aecb26f782aeb
I20250110 20:57:57.698488 7615 ts_tablet_manager.cc:1939] T d3a55ad10d794ff7bb7aecb26f782aeb P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:57.701642 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet d3a55ad10d794ff7bb7aecb26f782aeb (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:57.713956 7616 ts_tablet_manager.cc:1918] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:57.714315 7616 log.cc:1198] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/d3a55ad10d794ff7bb7aecb26f782aeb
I20250110 20:57:57.714953 7616 ts_tablet_manager.cc:1939] T d3a55ad10d794ff7bb7aecb26f782aeb P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:57.720845 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 143a38cabcea48e8926d9edcbe681def (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.721794 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143a38cabcea48e8926d9edcbe681def. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.727394 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 143a38cabcea48e8926d9edcbe681def (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.728339 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143a38cabcea48e8926d9edcbe681def. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.730329 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 143a38cabcea48e8926d9edcbe681def (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.731487 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143a38cabcea48e8926d9edcbe681def. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:57.739830 7617 ts_tablet_manager.cc:1918] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:57.740188 7617 log.cc:1198] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/d3a55ad10d794ff7bb7aecb26f782aeb
I20250110 20:57:57.740811 7617 ts_tablet_manager.cc:1939] T d3a55ad10d794ff7bb7aecb26f782aeb P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:57.742400 7606 tablet_bootstrap.cc:492] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:57.742954 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet d3a55ad10d794ff7bb7aecb26f782aeb (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:57.747328 7606 tablet_bootstrap.cc:654] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.759117 7606 tablet_bootstrap.cc:492] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:57.759526 7606 ts_tablet_manager.cc:1397] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.005s
I20250110 20:57:57.761713 7606 raft_consensus.cc:357] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.762385 7606 raft_consensus.cc:383] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.762655 7606 raft_consensus.cc:738] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.763217 7606 consensus_queue.cc:260] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.764300 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet d3a55ad10d794ff7bb7aecb26f782aeb (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:57.769074 7606 ts_tablet_manager.cc:1428] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20250110 20:57:57.777130 7609 tablet_bootstrap.cc:492] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:57.778982 7602 tablet_bootstrap.cc:492] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:57.783963 7609 tablet_bootstrap.cc:654] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:57.784538 7602 tablet_bootstrap.cc:654] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:57.800140 6980 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 6633402d9ab042acb750d47ad2dd70ee (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6633402d9ab042acb750d47ad2dd70ee consensus state has no leader
I20250110 20:57:57.811529 7609 tablet_bootstrap.cc:492] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:57.812009 7609 ts_tablet_manager.cc:1397] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.009s
I20250110 20:57:57.814539 7609 raft_consensus.cc:357] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.815449 7609 raft_consensus.cc:383] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.815745 7609 raft_consensus.cc:738] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.816526 7609 consensus_queue.cc:260] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.819167 7609 ts_tablet_manager.cc:1428] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
W20250110 20:57:57.823321 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:57.849895 7749 raft_consensus.cc:491] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:57.850317 7749 raft_consensus.cc:513] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.852921 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669acec9184749f094421b99bea0588f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:57.854161 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669acec9184749f094421b99bea0588f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:57.854758 7352 raft_consensus.cc:2463] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:57.853608 7208 raft_consensus.cc:2463] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:57.856195 7602 tablet_bootstrap.cc:492] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:57.856617 7602 ts_tablet_manager.cc:1397] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.078s user 0.010s sys 0.001s
I20250110 20:57:57.856653 7435 leader_election.cc:304] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:57.857520 7786 raft_consensus.cc:2798] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:57.857856 7786 raft_consensus.cc:491] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:57.858177 7786 raft_consensus.cc:3054] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:57.858924 7602 raft_consensus.cc:357] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.859524 7602 raft_consensus.cc:383] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:57.859760 7602 raft_consensus.cc:738] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:57.865538 7602 consensus_queue.cc:260] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.867993 7786 raft_consensus.cc:513] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.871096 7786 leader_election.cc:290] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:57.872103 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669acec9184749f094421b99bea0588f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:57.872627 7352 raft_consensus.cc:3054] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:57.873554 7749 leader_election.cc:290] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:57.882902 7352 raft_consensus.cc:2463] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:57.885243 7436 leader_election.cc:304] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:57.885946 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "669acec9184749f094421b99bea0588f" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:57.886529 7208 raft_consensus.cc:3054] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:57.889909 6980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250110 20:57:57.893002 6980 consensus_peers.cc:487] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 -> Peer f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203): Couldn't send request to peer f9bb61bdf30044a1b482d6a66e29435f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250110 20:57:57.887259 7749 raft_consensus.cc:2798] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:57.894451 7208 raft_consensus.cc:2463] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:57.895785 7749 raft_consensus.cc:695] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:57.896975 7602 ts_tablet_manager.cc:1428] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20250110 20:57:57.910254 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 143a38cabcea48e8926d9edcbe681def from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:57.922955 7749 consensus_queue.cc:237] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:57.928195 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 00f841d449b24c2ebe0c5cb708e4437d from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:57.930893 6986 catalog_manager.cc:2149] created HMS catalog entry for table [id=fc80a9112e4c4aa49f819343c2250505]
I20250110 20:57:57.932425 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 143a38cabcea48e8926d9edcbe681def
I20250110 20:57:57.940336 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 143a38cabcea48e8926d9edcbe681def with delete_type TABLET_DATA_DELETED (replaced by tablet 5efa6746553f4b308c923df89b67de32) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:57.940977 7615 tablet_replica.cc:331] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:57.941673 7615 raft_consensus.cc:2238] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.942090 7615 raft_consensus.cc:2267] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.944598 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 143a38cabcea48e8926d9edcbe681def with delete_type TABLET_DATA_DELETED (replaced by tablet 5efa6746553f4b308c923df89b67de32) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:57.945709 7617 tablet_replica.cc:331] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:57.946369 7617 raft_consensus.cc:2238] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.946789 7617 raft_consensus.cc:2267] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.948550 7617 ts_tablet_manager.cc:1905] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.955575 6986 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 11) took 4964 ms. Trace:
I20250110 20:57:57.945385 7615 ts_tablet_manager.cc:1905] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.955899 6986 rpcz_store.cc:276] 0110 20:57:52.991611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:52.995953 (+ 4342us) service_pool.cc:225] Handling call
0110 20:57:54.142815 (+1146862us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:54.144408 (+ 1593us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:57.930853 (+3786445us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:57:57.954911 (+ 24058us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:57.955265 (+ 354us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:57.955516 (+ 251us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:57.932688 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:57.933032 (+ 344us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:57.933051 (+ 19us) write_op.cc:435] Acquired schema lock
0110 20:57:57.933063 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:57.933330 (+ 267us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:57.933365 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:57.933386 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:57.933574 (+ 188us) tablet.cc:698] Row locks acquired
0110 20:57:57.933587 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:57.933686 (+ 99us) write_op.cc:270] Start()
0110 20:57:57.933756 (+ 70us) write_op.cc:276] Timestamp: P: 1736542677933616 usec, L: 0
0110 20:57:57.933770 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:57.934193 (+ 423us) log.cc:844] Serialized 432 byte log entry
0110 20:57:57.949492 (+ 15299us) op_driver.cc:464] REPLICATION: finished
0110 20:57:57.953637 (+ 4145us) write_op.cc:301] APPLY: starting
0110 20:57:57.953689 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:57.953782 (+ 93us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:57.953862 (+ 80us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:57.954210 (+ 348us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:57.954334 (+ 124us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:57.954373 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:57:57.954525 (+ 152us) log.cc:844] Serialized 28 byte log entry
0110 20:57:57.954633 (+ 108us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:57.954688 (+ 55us) write_op.cc:454] Released schema lock
0110 20:57:57.954715 (+ 27us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:57.954911 (+ 196us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:57.954934 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":3519034,"Hive Metastore.run_cpu_time_us":567,"Hive Metastore.run_wall_time_us":266307,"child_traces":[["op",{"apply.queue_time_us":3996,"apply.run_cpu_time_us":1244,"apply.run_wall_time_us":1244,"num_ops":2,"prepare.queue_time_us":146,"prepare.run_cpu_time_us":1798,"prepare.run_wall_time_us":1798,"raft.queue_time_us":825,"raft.run_cpu_time_us":577,"raft.run_wall_time_us":578,"replication_time_us":15621}]]}
I20250110 20:57:57.967475 6989 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 00f841d449b24c2ebe0c5cb708e4437d
I20250110 20:57:57.969048 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 143a38cabcea48e8926d9edcbe681def with delete_type TABLET_DATA_DELETED (replaced by tablet 5efa6746553f4b308c923df89b67de32) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:57.969784 7616 tablet_replica.cc:331] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:57.970448 7616 raft_consensus.cc:2238] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:57.970844 7616 raft_consensus.cc:2267] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:57.971413 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f841d449b24c2ebe0c5cb708e4437d with delete_type TABLET_DATA_DELETED (replaced by tablet f8632bafec7c408a92328b41ad9e4eef) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:57.972661 7616 ts_tablet_manager.cc:1905] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:57.974002 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f841d449b24c2ebe0c5cb708e4437d with delete_type TABLET_DATA_DELETED (replaced by tablet f8632bafec7c408a92328b41ad9e4eef) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:57.974725 6989 catalog_manager.cc:6596] ReplaceTablet: tablet 00f841d449b24c2ebe0c5cb708e4437d deleted and replaced by tablet f8632bafec7c408a92328b41ad9e4eef
I20250110 20:57:57.975231 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 00f841d449b24c2ebe0c5cb708e4437d with delete_type TABLET_DATA_DELETED (replaced by tablet f8632bafec7c408a92328b41ad9e4eef) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:57.981448 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 143a38cabcea48e8926d9edcbe681def deleted and replaced by tablet 5efa6746553f4b308c923df89b67de32
I20250110 20:57:57.992498 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 5efa6746553f4b308c923df89b67de32 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:57.993432 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efa6746553f4b308c923df89b67de32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.002516 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 5efa6746553f4b308c923df89b67de32 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.003494 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efa6746553f4b308c923df89b67de32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.005960 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 5efa6746553f4b308c923df89b67de32 (DEFAULT_TABLE table=default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.007031 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5efa6746553f4b308c923df89b67de32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.021847 7606 tablet_bootstrap.cc:492] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:58.026993 7606 tablet_bootstrap.cc:654] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.038753 7606 tablet_bootstrap.cc:492] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:58.039223 7606 ts_tablet_manager.cc:1397] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.002s
I20250110 20:57:58.048491 6988 catalog_manager.cc:2149] created HMS catalog entry for table [id=ca7388d37c1f4aa8ac6ff138c21c7047]
I20250110 20:57:58.050532 7606 raft_consensus.cc:357] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.053606 7606 raft_consensus.cc:383] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.054198 7606 raft_consensus.cc:738] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.065176 6990 catalog_manager.cc:5526] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:58.070238 7606 consensus_queue.cc:260] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.071933 6990 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:58.073362 7606 ts_tablet_manager.cc:1428] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.034s user 0.003s sys 0.001s
I20250110 20:57:58.092136 7617 ts_tablet_manager.cc:1918] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.092504 7617 log.cc:1198] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/143a38cabcea48e8926d9edcbe681def
I20250110 20:57:58.093120 7617 ts_tablet_manager.cc:1939] T 143a38cabcea48e8926d9edcbe681def P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:58.095599 6988 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 12) took 5091 ms. Trace:
I20250110 20:57:58.095914 6988 rpcz_store.cc:276] 0110 20:57:53.004282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:53.005014 (+ 732us) service_pool.cc:225] Handling call
0110 20:57:54.147392 (+1142378us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:54.149194 (+ 1802us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:58.048448 (+3899254us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:57:58.094969 (+ 46521us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:58.095286 (+ 317us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:58.095541 (+ 255us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:58.059471 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:58.060083 (+ 612us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:58.060107 (+ 24us) write_op.cc:435] Acquired schema lock
0110 20:57:58.060120 (+ 13us) tablet.cc:659] Decoding operations
0110 20:57:58.060369 (+ 249us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:58.060399 (+ 30us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:58.060420 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:58.060581 (+ 161us) tablet.cc:698] Row locks acquired
0110 20:57:58.060593 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:58.060684 (+ 91us) write_op.cc:270] Start()
0110 20:57:58.060752 (+ 68us) write_op.cc:276] Timestamp: P: 1736542678060604 usec, L: 0
0110 20:57:58.060766 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:58.061165 (+ 399us) log.cc:844] Serialized 432 byte log entry
0110 20:57:58.080662 (+ 19497us) op_driver.cc:464] REPLICATION: finished
0110 20:57:58.087553 (+ 6891us) write_op.cc:301] APPLY: starting
0110 20:57:58.087605 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:58.087697 (+ 92us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:58.087776 (+ 79us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:58.088112 (+ 336us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:58.088235 (+ 123us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:58.088273 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:57:58.088427 (+ 154us) log.cc:844] Serialized 28 byte log entry
0110 20:57:58.088559 (+ 132us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:58.094697 (+ 6138us) write_op.cc:454] Released schema lock
0110 20:57:58.094735 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:58.094984 (+ 249us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:58.095012 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":3779858,"Hive Metastore.run_cpu_time_us":624,"Hive Metastore.run_wall_time_us":117790,"child_traces":[["op",{"apply.queue_time_us":6746,"apply.run_cpu_time_us":1323,"apply.run_wall_time_us":7391,"num_ops":2,"prepare.queue_time_us":9220,"prepare.run_cpu_time_us":1642,"prepare.run_wall_time_us":2068,"raft.queue_time_us":74,"raft.run_cpu_time_us":11,"raft.run_wall_time_us":11,"replication_time_us":19811}]]}
I20250110 20:57:58.112049 7615 ts_tablet_manager.cc:1918] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.112525 7615 log.cc:1198] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/143a38cabcea48e8926d9edcbe681def
I20250110 20:57:58.113116 7615 ts_tablet_manager.cc:1939] T 143a38cabcea48e8926d9edcbe681def P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:58.115346 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 143a38cabcea48e8926d9edcbe681def (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:58.116273 7602 tablet_bootstrap.cc:492] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:58.121281 7602 tablet_bootstrap.cc:654] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.129374 7616 ts_tablet_manager.cc:1918] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.129763 7616 log.cc:1198] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/143a38cabcea48e8926d9edcbe681def
I20250110 20:57:58.130370 7616 ts_tablet_manager.cc:1939] T 143a38cabcea48e8926d9edcbe681def P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:58.132910 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 143a38cabcea48e8926d9edcbe681def (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:58.134960 7609 tablet_bootstrap.cc:492] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:58.135382 7602 tablet_bootstrap.cc:492] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:58.135748 7602 ts_tablet_manager.cc:1397] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20250110 20:57:58.143728 7602 raft_consensus.cc:357] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.144508 7602 raft_consensus.cc:383] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.144821 7602 raft_consensus.cc:738] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.145648 7602 consensus_queue.cc:260] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.149744 7602 ts_tablet_manager.cc:1428] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20250110 20:57:58.162335 7789 raft_consensus.cc:491] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:58.162770 7789 raft_consensus.cc:513] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.164597 7789 leader_election.cc:290] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:58.165920 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f841d449b24c2ebe0c5cb708e4437d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:58.166455 7208 raft_consensus.cc:2463] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:58.168457 7435 leader_election.cc:304] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:58.169240 7789 raft_consensus.cc:2798] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:58.169621 7789 raft_consensus.cc:491] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:58.170176 7789 raft_consensus.cc:3054] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:58.169795 6986 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_1073e5a3e4b84cbc9198aa13f15af95e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:58.170965 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f841d449b24c2ebe0c5cb708e4437d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:58.171646 7353 raft_consensus.cc:2463] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:58.174198 7617 tablet_replica.cc:331] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:58.174260 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 143a38cabcea48e8926d9edcbe681def (table default.table_a1cbfe0292df4dc8a2c9b8709f0df836 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:58.174943 7617 raft_consensus.cc:2238] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.175367 7617 raft_consensus.cc:2267] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.177081 7617 ts_tablet_manager.cc:1905] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.187204 7609 tablet_bootstrap.cc:654] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:58.192768 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:58.194159 7795 raft_consensus.cc:491] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:58.194568 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 25b2023ef65a4cceadc07f05f1839309 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:58.195048 7795 raft_consensus.cc:513] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.198976 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efa6746553f4b308c923df89b67de32" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:58.199247 7615 tablet_replica.cc:331] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:58.199580 7208 raft_consensus.cc:2463] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:57:58.200016 7615 raft_consensus.cc:2238] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.200454 7615 raft_consensus.cc:2267] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.201792 7616 tablet_replica.cc:331] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:58.204973 7287 leader_election.cc:304] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:58.210711 7615 ts_tablet_manager.cc:1905] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.210871 7796 raft_consensus.cc:2798] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:58.211205 7796 raft_consensus.cc:491] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:58.211503 7796 raft_consensus.cc:3054] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:58.230198 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 25b2023ef65a4cceadc07f05f1839309
W20250110 20:57:58.230262 7049 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5efa6746553f4b308c923df89b67de32 consensus state has no leader
I20250110 20:57:58.231259 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4a533c92a24b5da58a75886c417f47 (DEFAULT_TABLE table=default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.232237 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4a533c92a24b5da58a75886c417f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.233120 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b2023ef65a4cceadc07f05f1839309 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d2fb488db4247b3bad2a4905d160fbd) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:58.233963 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4a533c92a24b5da58a75886c417f47 (DEFAULT_TABLE table=default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.234210 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:58.234890 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4a533c92a24b5da58a75886c417f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.239699 7774 consensus_queue.cc:1035] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:58.240540 7049 catalog_manager.cc:2710] Deleting table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d] in response to Hive Metastore notification log event 8
I20250110 20:57:58.244485 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4a533c92a24b5da58a75886c417f47 (DEFAULT_TABLE table=default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.245431 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4a533c92a24b5da58a75886c417f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.247573 6989 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:58.255023 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 25b2023ef65a4cceadc07f05f1839309 deleted and replaced by tablet 1d2fb488db4247b3bad2a4905d160fbd
I20250110 20:57:58.255306 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b2023ef65a4cceadc07f05f1839309 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d2fb488db4247b3bad2a4905d160fbd) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:58.257377 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b2023ef65a4cceadc07f05f1839309 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d2fb488db4247b3bad2a4905d160fbd) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:58.260038 7189 tablet_service.cc:1467] Processing CreateTablet for tablet f8632bafec7c408a92328b41ad9e4eef (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.260955 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8632bafec7c408a92328b41ad9e4eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.260663 7796 raft_consensus.cc:513] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.263275 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efa6746553f4b308c923df89b67de32" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:58.263789 7208 raft_consensus.cc:3054] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:58.269353 7332 tablet_service.cc:1467] Processing CreateTablet for tablet f8632bafec7c408a92328b41ad9e4eef (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.270319 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8632bafec7c408a92328b41ad9e4eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.274555 7477 tablet_service.cc:1467] Processing CreateTablet for tablet f8632bafec7c408a92328b41ad9e4eef (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.275465 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8632bafec7c408a92328b41ad9e4eef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.275501 7602 tablet_bootstrap.cc:492] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:58.282788 7602 tablet_bootstrap.cc:654] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.290335 7602 tablet_bootstrap.cc:492] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:58.290741 7602 ts_tablet_manager.cc:1397] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20250110 20:57:58.301726 7208 raft_consensus.cc:2463] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:58.302826 7287 leader_election.cc:304] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:58.303612 7762 raft_consensus.cc:2798] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:58.305487 6989 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_35a55b6fae704e909310ca9089810b45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:58.315400 7049 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 30aa8fd175de4c3aac137ac51f1ae51d
I20250110 20:57:58.292737 7602 raft_consensus.cc:357] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.317963 7602 raft_consensus.cc:383] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.318220 7602 raft_consensus.cc:738] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.196395 7789 raft_consensus.cc:513] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.318858 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 30aa8fd175de4c3aac137ac51f1ae51d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:58 UTC) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:58.318914 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 30aa8fd175de4c3aac137ac51f1ae51d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:58 UTC) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:58.320515 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efa6746553f4b308c923df89b67de32" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
W20250110 20:57:58.321821 7288 leader_election.cc:343] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:58.323047 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 30aa8fd175de4c3aac137ac51f1ae51d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:58 UTC) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:58.323683 7616 raft_consensus.cc:2238] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.323737 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5efa6746553f4b308c923df89b67de32" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:58.324558 7616 raft_consensus.cc:2267] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:58.325253 7288 leader_election.cc:343] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:58.324833 7602 consensus_queue.cc:260] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.331007 7796 leader_election.cc:290] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:58.333792 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f841d449b24c2ebe0c5cb708e4437d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
W20250110 20:57:58.335014 7435 leader_election.cc:343] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:58.335709 7795 leader_election.cc:290] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:58.339962 7609 tablet_bootstrap.cc:492] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:58.340366 7609 ts_tablet_manager.cc:1397] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.206s user 0.014s sys 0.005s
I20250110 20:57:58.345542 7609 raft_consensus.cc:357] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.346215 7609 raft_consensus.cc:383] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.346457 7609 raft_consensus.cc:738] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.347649 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00f841d449b24c2ebe0c5cb708e4437d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
W20250110 20:57:58.348971 7436 leader_election.cc:343] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:58.349347 7436 leader_election.cc:304] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
W20250110 20:57:58.349893 7436 raft_consensus.cc:2681] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250110 20:57:58.354547 6985 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_df681f6d6d9e483282a9b28e42817aef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:58.353531 7789 leader_election.cc:290] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:58.357398 7609 consensus_queue.cc:260] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.359115 7762 raft_consensus.cc:695] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:58.360811 6994 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33774 (request call id 5) took 5647 ms. Trace:
I20250110 20:57:58.361555 7616 ts_tablet_manager.cc:1905] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.361120 6994 rpcz_store.cc:276] 0110 20:57:52.713723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:52.716579 (+ 2856us) service_pool.cc:225] Handling call
0110 20:57:53.086139 (+369560us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:58.357647 (+5271508us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:58.359905 (+ 2258us) catalog_manager.cc:3627] Apply alter schema
0110 20:57:58.360732 (+ 827us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":197,"Hive Metastore.run_cpu_time_us":2255,"Hive Metastore.run_wall_time_us":1017940}
I20250110 20:57:58.370193 7800 consensus_queue.cc:1035] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:58.359907 7762 consensus_queue.cc:237] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.375766 7609 ts_tablet_manager.cc:1428] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20250110 20:57:58.376574 7609 tablet_bootstrap.cc:492] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:58.383172 7615 ts_tablet_manager.cc:1918] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.387223 7609 tablet_bootstrap.cc:654] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.397658 7606 tablet_bootstrap.cc:492] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:58.403471 7617 ts_tablet_manager.cc:1918] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.403858 7617 log.cc:1198] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/00f841d449b24c2ebe0c5cb708e4437d
I20250110 20:57:58.404067 7606 tablet_bootstrap.cc:654] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.404429 7617 ts_tablet_manager.cc:1939] T 00f841d449b24c2ebe0c5cb708e4437d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:58.406653 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 00f841d449b24c2ebe0c5cb708e4437d (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:58.407382 7615 log.cc:1198] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/00f841d449b24c2ebe0c5cb708e4437d
I20250110 20:57:58.408018 7615 ts_tablet_manager.cc:1939] T 00f841d449b24c2ebe0c5cb708e4437d P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:58.410148 7615 tablet_replica.cc:331] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:58.410326 6994 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:58.400043 7602 ts_tablet_manager.cc:1428] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.109s user 0.008s sys 0.000s
I20250110 20:57:58.413189 7602 tablet_bootstrap.cc:492] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:58.415208 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 00f841d449b24c2ebe0c5cb708e4437d (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:58.417124 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fb488db4247b3bad2a4905d160fbd (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.418126 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2fb488db4247b3bad2a4905d160fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.422796 7602 tablet_bootstrap.cc:654] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.427969 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fb488db4247b3bad2a4905d160fbd (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.428891 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2fb488db4247b3bad2a4905d160fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.429734 6988 catalog_manager.cc:5526] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:58.431131 6988 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:58.444188 7352 raft_consensus.cc:1270] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Refusing update from remote peer c3b63d95ca734d8a9803b96bfd94f307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:58.446787 7606 tablet_bootstrap.cc:492] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:58.447191 7606 ts_tablet_manager.cc:1397] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.003s
I20250110 20:57:58.448268 7615 raft_consensus.cc:2238] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:58.448987 7615 raft_consensus.cc:2267] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.449640 7208 raft_consensus.cc:1270] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer c3b63d95ca734d8a9803b96bfd94f307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:58.449841 7617 tablet_replica.cc:331] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:58.450912 7789 consensus_queue.cc:1035] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:58.451581 7749 consensus_queue.cc:1035] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:58.451957 7615 ts_tablet_manager.cc:1905] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.453974 6994 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:58.467303 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 25b2023ef65a4cceadc07f05f1839309 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:58.467741 7606 raft_consensus.cc:357] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.468431 7606 raft_consensus.cc:383] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.468667 7606 raft_consensus.cc:738] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.472556 7803 mvcc.cc:204] Tried to move back new op lower bound from 7112878810889805824 to 7112878808857456640. Current Snapshot: MvccSnapshot[applied={T|T < 7112878810889805824}]
I20250110 20:57:58.478068 7615 ts_tablet_manager.cc:1918] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:58.478400 7615 log.cc:1198] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/25b2023ef65a4cceadc07f05f1839309
I20250110 20:57:58.479028 7615 ts_tablet_manager.cc:1939] T 25b2023ef65a4cceadc07f05f1839309 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:58.481673 7602 tablet_bootstrap.cc:492] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:58.482072 7602 ts_tablet_manager.cc:1397] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.002s
I20250110 20:57:58.484735 7602 raft_consensus.cc:357] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.485342 7602 raft_consensus.cc:383] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.485154 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:58.485594 7602 raft_consensus.cc:738] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.486431 7615 tablet_replica.cc:331] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:58.486140 7602 consensus_queue.cc:260] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.487926 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 25b2023ef65a4cceadc07f05f1839309 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:58.487073 7615 raft_consensus.cc:2238] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.494181 7615 raft_consensus.cc:2267] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.495812 7615 ts_tablet_manager.cc:1905] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.498696 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 02b0bf75f58e4a02a8c7def0bff5119f (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.499617 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b0bf75f58e4a02a8c7def0bff5119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.503428 7602 ts_tablet_manager.cc:1428] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250110 20:57:58.504241 7602 tablet_bootstrap.cc:492] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:58.511217 7814 tablet.cc:1721] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:58.509374 7602 tablet_bootstrap.cc:654] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.515182 7815 tablet.cc:1721] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:58.527931 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:58.531528 7615 ts_tablet_manager.cc:1918] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:58.531880 7615 log.cc:1198] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/30aa8fd175de4c3aac137ac51f1ae51d
I20250110 20:57:58.532524 7615 ts_tablet_manager.cc:1939] T 30aa8fd175de4c3aac137ac51f1ae51d P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:58.469233 7606 consensus_queue.cc:260] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.535585 7606 ts_tablet_manager.cc:1428] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.088s user 0.005s sys 0.001s
I20250110 20:57:58.536398 7606 tablet_bootstrap.cc:492] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:58.544440 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 30aa8fd175de4c3aac137ac51f1ae51d (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:58.546979 7617 raft_consensus.cc:2238] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.547883 7617 raft_consensus.cc:2267] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.550235 7617 ts_tablet_manager.cc:1905] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.559414 7602 tablet_bootstrap.cc:492] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:58.559804 7602 ts_tablet_manager.cc:1397] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.008s
I20250110 20:57:58.561600 7616 ts_tablet_manager.cc:1918] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.562062 7616 log.cc:1198] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/00f841d449b24c2ebe0c5cb708e4437d
I20250110 20:57:58.562726 7616 ts_tablet_manager.cc:1939] T 00f841d449b24c2ebe0c5cb708e4437d P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:58.563575 7602 raft_consensus.cc:357] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.564200 7602 raft_consensus.cc:383] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.564442 7602 raft_consensus.cc:738] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.564986 7602 consensus_queue.cc:260] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.571322 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 02b0bf75f58e4a02a8c7def0bff5119f (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.572388 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b0bf75f58e4a02a8c7def0bff5119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.575903 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2fb488db4247b3bad2a4905d160fbd (DEFAULT_TABLE table=default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.576833 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d2fb488db4247b3bad2a4905d160fbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.582010 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 02b0bf75f58e4a02a8c7def0bff5119f (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.582906 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02b0bf75f58e4a02a8c7def0bff5119f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.592563 7602 ts_tablet_manager.cc:1428] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.032s user 0.006s sys 0.002s
I20250110 20:57:58.593700 7602 tablet_bootstrap.cc:492] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:58.598444 7616 tablet_replica.cc:331] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:58.599246 7616 raft_consensus.cc:2238] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.600343 7602 tablet_bootstrap.cc:654] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.601229 7819 tablet.cc:1721] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:58.606575 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 00f841d449b24c2ebe0c5cb708e4437d (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:58.608251 7609 tablet_bootstrap.cc:492] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:58.608628 7609 ts_tablet_manager.cc:1397] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.232s user 0.033s sys 0.034s
I20250110 20:57:58.612172 6988 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_045d0574736e4b9e8fdf9e6a91066d06" } new_table_name: "default.Table_176fb71364f24095a1efbf0d29c18ff8" modify_external_catalogs: true
I20250110 20:57:58.615872 6994 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:58.616307 7609 raft_consensus.cc:357] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.616983 7609 raft_consensus.cc:383] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.617234 7609 raft_consensus.cc:738] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.617808 7609 consensus_queue.cc:260] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.620267 7609 ts_tablet_manager.cc:1428] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20250110 20:57:58.621788 7609 tablet_bootstrap.cc:492] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:58.623724 7606 tablet_bootstrap.cc:654] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.626073 7602 tablet_bootstrap.cc:492] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:58.626519 7602 ts_tablet_manager.cc:1397] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.033s user 0.013s sys 0.002s
I20250110 20:57:58.628959 7602 raft_consensus.cc:357] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.629841 7616 raft_consensus.cc:2267] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.630055 7602 raft_consensus.cc:383] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.630391 7602 raft_consensus.cc:738] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.631006 7602 consensus_queue.cc:260] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.644289 7609 tablet_bootstrap.cc:654] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.650697 7617 ts_tablet_manager.cc:1918] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:58.652644 7617 log.cc:1198] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/25b2023ef65a4cceadc07f05f1839309
I20250110 20:57:58.653190 7616 ts_tablet_manager.cc:1905] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.653307 7617 ts_tablet_manager.cc:1939] T 25b2023ef65a4cceadc07f05f1839309 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:58.655472 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 25b2023ef65a4cceadc07f05f1839309 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:58.672448 7616 ts_tablet_manager.cc:1918] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:58.672688 7617 tablet_replica.cc:331] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:58.672799 7616 log.cc:1198] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/25b2023ef65a4cceadc07f05f1839309
I20250110 20:57:58.673159 6994 catalog_manager.cc:6071] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f] alter complete (version 1)
I20250110 20:57:58.673336 7617 raft_consensus.cc:2238] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:58.673470 7616 ts_tablet_manager.cc:1939] T 25b2023ef65a4cceadc07f05f1839309 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:58.675284 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 25b2023ef65a4cceadc07f05f1839309 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d2fb488db4247b3bad2a4905d160fbd) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:58.681905 7606 tablet_bootstrap.cc:492] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:58.682317 7606 ts_tablet_manager.cc:1397] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.146s user 0.009s sys 0.000s
I20250110 20:57:58.683459 7501 raft_consensus.cc:3054] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:58.684641 7602 ts_tablet_manager.cc:1428] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.058s user 0.005s sys 0.000s
I20250110 20:57:58.685176 7617 raft_consensus.cc:2267] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.689000 7208 raft_consensus.cc:1270] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:58.690140 7501 raft_consensus.cc:1270] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:57:58.691488 7801 consensus_queue.cc:1035] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:57:58.694245 7802 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:58.694648 7802 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.696419 7802 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:58.702474 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b0bf75f58e4a02a8c7def0bff5119f" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:58.702880 7801 consensus_queue.cc:1035] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:57:58.704110 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b0bf75f58e4a02a8c7def0bff5119f" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:58.702473 7617 ts_tablet_manager.cc:1905] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:58.708005 7145 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:57:58.708525 7145 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:58.708843 7145 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:58.709699 7802 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:58.709276 7606 raft_consensus.cc:357] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.711066 7606 raft_consensus.cc:383] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.716519 7609 tablet_bootstrap.cc:492] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:58.716913 7609 ts_tablet_manager.cc:1397] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.095s user 0.010s sys 0.000s
I20250110 20:57:58.719076 7609 raft_consensus.cc:357] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.719693 7609 raft_consensus.cc:383] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.719941 7609 raft_consensus.cc:738] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.720902 6994 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:58.711314 7606 raft_consensus.cc:738] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.722911 7606 consensus_queue.cc:260] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.731861 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 25b2023ef65a4cceadc07f05f1839309 (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:58.738445 7609 consensus_queue.cc:260] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.739526 6994 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:58.743024 7819 tablet.cc:1721] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:58.760530 7616 tablet_replica.cc:331] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:58.761221 7616 raft_consensus.cc:2238] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.761628 7606 ts_tablet_manager.cc:1428] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.079s user 0.007s sys 0.000s
I20250110 20:57:58.762476 7606 tablet_bootstrap.cc:492] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:58.761814 7815 tablet.cc:1721] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:58.767431 7814 tablet.cc:1721] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:57:58.754014 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 3d4a533c92a24b5da58a75886c417f47 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:58.789029 7609 ts_tablet_manager.cc:1428] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.072s user 0.002s sys 0.004s
I20250110 20:57:58.790916 7609 tablet_bootstrap.cc:492] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:58.797547 7616 raft_consensus.cc:2267] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.798161 7617 ts_tablet_manager.cc:1918] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:58.798534 7617 log.cc:1198] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/30aa8fd175de4c3aac137ac51f1ae51d
I20250110 20:57:58.799549 7617 ts_tablet_manager.cc:1939] T 30aa8fd175de4c3aac137ac51f1ae51d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:58.801975 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 30aa8fd175de4c3aac137ac51f1ae51d (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:58.802541 7616 ts_tablet_manager.cc:1905] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:57:58.803284 6980 catalog_manager.cc:4839] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet 25b2023ef65a4cceadc07f05f1839309 because the tablet was not found. No further retry: Not found: Tablet not found: 25b2023ef65a4cceadc07f05f1839309
I20250110 20:57:58.803618 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 3d4a533c92a24b5da58a75886c417f47
I20250110 20:57:58.806382 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4a533c92a24b5da58a75886c417f47 with delete_type TABLET_DATA_DELETED (replaced by tablet e73e60dfdab34bee95fd3bfc2fede7be) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:58.807015 7617 tablet_replica.cc:331] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:58.807607 7617 raft_consensus.cc:2238] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.807992 7617 raft_consensus.cc:2267] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.812603 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4a533c92a24b5da58a75886c417f47 with delete_type TABLET_DATA_DELETED (replaced by tablet e73e60dfdab34bee95fd3bfc2fede7be) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:58.813107 7617 ts_tablet_manager.cc:1905] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.814105 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 3d4a533c92a24b5da58a75886c417f47 deleted and replaced by tablet e73e60dfdab34bee95fd3bfc2fede7be
I20250110 20:57:58.814491 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4a533c92a24b5da58a75886c417f47 with delete_type TABLET_DATA_DELETED (replaced by tablet e73e60dfdab34bee95fd3bfc2fede7be) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:58.815114 7615 tablet_replica.cc:331] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:58.815796 7615 raft_consensus.cc:2238] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.816198 7615 raft_consensus.cc:2267] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250110 20:57:58.815439 6980 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 5a955851581349debfc895eed2411635 (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5a955851581349debfc895eed2411635 consensus state has no leader
I20250110 20:57:58.818936 7615 ts_tablet_manager.cc:1905] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.824914 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:58.829883 7609 tablet_bootstrap.cc:654] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.837177 6994 catalog_manager.cc:6071] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401] alter complete (version 1)
I20250110 20:57:58.838335 7787 raft_consensus.cc:491] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:58.838745 7787 raft_consensus.cc:513] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.841646 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4a533c92a24b5da58a75886c417f47" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:58.841784 7787 leader_election.cc:290] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:57:58.842900 7435 leader_election.cc:343] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:58.854032 7606 tablet_bootstrap.cc:654] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.857616 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4a533c92a24b5da58a75886c417f47" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
W20250110 20:57:58.859977 7436 leader_election.cc:343] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: Tablet not RUNNING: STOPPED
I20250110 20:57:58.860605 7436 leader_election.cc:304] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:57:58.861297 7787 raft_consensus.cc:2743] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:57:58.862196 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:57:58.862521 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:58.866096 7190 tablet_service.cc:1467] Processing CreateTablet for tablet e73e60dfdab34bee95fd3bfc2fede7be (DEFAULT_TABLE table=default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.866982 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73e60dfdab34bee95fd3bfc2fede7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.867898 7617 ts_tablet_manager.cc:1918] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.868219 7617 log.cc:1198] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/3d4a533c92a24b5da58a75886c417f47
I20250110 20:57:58.868749 7617 ts_tablet_manager.cc:1939] T 3d4a533c92a24b5da58a75886c417f47 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:58.875980 7615 ts_tablet_manager.cc:1918] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:58.876295 7615 log.cc:1198] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/3d4a533c92a24b5da58a75886c417f47
I20250110 20:57:58.876852 7615 ts_tablet_manager.cc:1939] T 3d4a533c92a24b5da58a75886c417f47 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:58.877218 7606 tablet_bootstrap.cc:492] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:58.877622 7606 ts_tablet_manager.cc:1397] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.002s
I20250110 20:57:58.878449 7331 tablet_service.cc:1467] Processing CreateTablet for tablet e73e60dfdab34bee95fd3bfc2fede7be (DEFAULT_TABLE table=default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.879405 6993 catalog_manager.cc:2686] deleted HMS catalog entry for table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]
I20250110 20:57:58.881753 7477 tablet_service.cc:1467] Processing CreateTablet for tablet e73e60dfdab34bee95fd3bfc2fede7be (DEFAULT_TABLE table=default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:58.882709 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73e60dfdab34bee95fd3bfc2fede7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:58.882449 7606 raft_consensus.cc:357] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.883607 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 3d4a533c92a24b5da58a75886c417f47 (table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]) successfully deleted
I20250110 20:57:58.884383 7609 tablet_bootstrap.cc:492] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:58.884748 7609 ts_tablet_manager.cc:1397] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.094s user 0.001s sys 0.009s
I20250110 20:57:58.886893 7609 raft_consensus.cc:357] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.887578 7609 raft_consensus.cc:383] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.887815 7609 raft_consensus.cc:738] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.889322 7602 tablet_bootstrap.cc:492] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:58.891600 7616 ts_tablet_manager.cc:1918] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:58.891957 7616 log.cc:1198] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/30aa8fd175de4c3aac137ac51f1ae51d
I20250110 20:57:58.892648 7616 ts_tablet_manager.cc:1939] T 30aa8fd175de4c3aac137ac51f1ae51d P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:58.883090 7606 raft_consensus.cc:383] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.893024 7606 raft_consensus.cc:738] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.893661 7606 consensus_queue.cc:260] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.892287 7609 consensus_queue.cc:260] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:58.894829 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 30aa8fd175de4c3aac137ac51f1ae51d (table default.table_c8e132d9095947f8b4700839d70d8060 [id=d3ea810c260f466cab0c3f5edb25366d]) successfully deleted
I20250110 20:57:58.894981 7602 tablet_bootstrap.cc:654] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.899550 7616 tablet_replica.cc:331] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:58.902604 7609 ts_tablet_manager.cc:1428] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20250110 20:57:58.903010 7616 raft_consensus.cc:2238] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 20:57:58.903192 6987 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_dca3421fe79d4330bc146c208b21a6f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:58.903445 7616 raft_consensus.cc:2267] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.903738 7606 ts_tablet_manager.cc:1428] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20250110 20:57:58.904529 7606 tablet_bootstrap.cc:492] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:58.904448 6991 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33774 (request call id 6) took 6174 ms. Trace:
I20250110 20:57:58.905601 6992 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33774 (request call id 7) took 6102 ms. Trace:
I20250110 20:57:58.906138 7616 ts_tablet_manager.cc:1905] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.905916 6992 rpcz_store.cc:276] 0110 20:57:52.803299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:52.803435 (+ 136us) service_pool.cc:225] Handling call
0110 20:57:54.141160 (+1337725us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:58.905533 (+4764373us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":551334,"Hive Metastore.run_cpu_time_us":773,"Hive Metastore.run_wall_time_us":2970532}
I20250110 20:57:58.907929 7609 tablet_bootstrap.cc:492] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
W20250110 20:57:58.907905 6986 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_1073e5a3e4b84cbc9198aa13f15af95e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250110 20:57:58.909070 6989 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_35a55b6fae704e909310ca9089810b45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:57:58.912608 7787 raft_consensus.cc:491] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:58.913017 7787 raft_consensus.cc:513] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.914855 7787 leader_election.cc:290] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:58.916556 7606 tablet_bootstrap.cc:654] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.916371 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:58.916954 7208 raft_consensus.cc:2463] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:58.904762 6991 rpcz_store.cc:276] 0110 20:57:52.729750 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:52.729871 (+ 121us) service_pool.cc:225] Handling call
0110 20:57:54.140127 (+1410256us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:58.902638 (+4762511us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:58.903588 (+ 950us) catalog_manager.cc:3627] Apply alter schema
0110 20:57:58.904394 (+ 806us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1033,"Hive Metastore.run_cpu_time_us":1622,"Hive Metastore.run_wall_time_us":379438}
I20250110 20:57:58.920778 7435 leader_election.cc:304] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:58.921736 7609 tablet_bootstrap.cc:654] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:58.921691 7787 raft_consensus.cc:2798] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:58.922063 7787 raft_consensus.cc:491] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:58.922353 7787 raft_consensus.cc:3054] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:58.930445 6994 master_service.cc:946] ReplaceTablet: received request to replace tablet 02b0bf75f58e4a02a8c7def0bff5119f from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:58.934772 7787 raft_consensus.cc:513] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.938553 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:58.939107 7208 raft_consensus.cc:3054] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:58.945618 7208 raft_consensus.cc:2463] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:58.946666 7435 leader_election.cc:304] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:58.947541 7825 raft_consensus.cc:2798] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:58.948001 7825 raft_consensus.cc:695] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:58.948686 7825 consensus_queue.cc:237] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.949970 7602 tablet_bootstrap.cc:492] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:58.950366 7602 ts_tablet_manager.cc:1397] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.005s
I20250110 20:57:58.953292 7602 raft_consensus.cc:357] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.953933 7602 raft_consensus.cc:383] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:58.954181 7602 raft_consensus.cc:738] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:58.954699 7602 consensus_queue.cc:260] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.959169 7787 leader_election.cc:290] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:58.959538 7602 ts_tablet_manager.cc:1428] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250110 20:57:58.982991 6994 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:57:58.986119 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b0bf75f58e4a02a8c7def0bff5119f with delete_type TABLET_DATA_DELETED (replaced by tablet c194735eea7847e1a6abbbdcf34e9579) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:58.986122 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b0bf75f58e4a02a8c7def0bff5119f with delete_type TABLET_DATA_DELETED (replaced by tablet c194735eea7847e1a6abbbdcf34e9579) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:58.986704 6994 catalog_manager.cc:6596] ReplaceTablet: tablet 02b0bf75f58e4a02a8c7def0bff5119f deleted and replaced by tablet c194735eea7847e1a6abbbdcf34e9579
I20250110 20:57:58.986910 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b0bf75f58e4a02a8c7def0bff5119f with delete_type TABLET_DATA_DELETED (replaced by tablet c194735eea7847e1a6abbbdcf34e9579) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:58.987507 7615 tablet_replica.cc:331] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:58.988160 7615 raft_consensus.cc:2238] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:58.988561 7615 raft_consensus.cc:2267] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:58.990845 7615 ts_tablet_manager.cc:1905] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:58.994480 7802 raft_consensus.cc:491] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:58.994897 7802 raft_consensus.cc:513] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:58.996608 7802 leader_election.cc:290] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:59.000367 7609 tablet_bootstrap.cc:492] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:59.000757 7609 ts_tablet_manager.cc:1397] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.093s user 0.015s sys 0.004s
I20250110 20:57:59.004302 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e73e60dfdab34bee95fd3bfc2fede7be" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
W20250110 20:57:59.005486 7145 leader_election.cc:343] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:59.003147 7609 raft_consensus.cc:357] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.008508 7609 raft_consensus.cc:383] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.008761 7609 raft_consensus.cc:738] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.014750 6991 catalog_manager.cc:5526] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:59.015242 7609 consensus_queue.cc:260] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.017164 7609 ts_tablet_manager.cc:1428] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20250110 20:57:59.021373 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 3d4a533c92a24b5da58a75886c417f47 (table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]) successfully deleted
I20250110 20:57:59.024322 7616 ts_tablet_manager.cc:1918] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:59.024727 7616 log.cc:1198] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/3d4a533c92a24b5da58a75886c417f47
I20250110 20:57:59.025369 7616 ts_tablet_manager.cc:1939] T 3d4a533c92a24b5da58a75886c417f47 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:59.029875 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b0bf75f58e4a02a8c7def0bff5119f with delete_type TABLET_DATA_DELETED (replaced by tablet c194735eea7847e1a6abbbdcf34e9579) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.031004 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e73e60dfdab34bee95fd3bfc2fede7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.030251 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:59.031735 7353 raft_consensus.cc:2463] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:59.032788 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:59.033345 7352 raft_consensus.cc:3054] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:57:59.038501 7145 leader_election.cc:343] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: e73e60dfdab34bee95fd3bfc2fede7be
I20250110 20:57:59.039619 7145 leader_election.cc:304] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:59.040303 7802 raft_consensus.cc:2743] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:59.041009 7332 tablet_service.cc:1467] Processing CreateTablet for tablet c194735eea7847e1a6abbbdcf34e9579 (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.042169 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c194735eea7847e1a6abbbdcf34e9579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.045099 7615 ts_tablet_manager.cc:1918] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:59.045612 6994 master_service.cc:946] ReplaceTablet: received request to replace tablet 5efa6746553f4b308c923df89b67de32 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:59.045858 7615 log.cc:1198] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:57:59.046463 7615 ts_tablet_manager.cc:1939] T 02b0bf75f58e4a02a8c7def0bff5119f P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:59.057279 7477 tablet_service.cc:1467] Processing CreateTablet for tablet c194735eea7847e1a6abbbdcf34e9579 (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.058317 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c194735eea7847e1a6abbbdcf34e9579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.048523 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 02b0bf75f58e4a02a8c7def0bff5119f (table default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:57:59.068429 7352 raft_consensus.cc:2463] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.072206 7609 tablet_bootstrap.cc:492] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:59.076299 7616 tablet_replica.cc:331] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:59.076179 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 3d4a533c92a24b5da58a75886c417f47 (table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]) successfully deleted
I20250110 20:57:59.076962 7616 raft_consensus.cc:2238] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.077380 7616 raft_consensus.cc:2267] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.079346 7190 tablet_service.cc:1467] Processing CreateTablet for tablet c194735eea7847e1a6abbbdcf34e9579 (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.080365 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c194735eea7847e1a6abbbdcf34e9579. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.083192 7609 tablet_bootstrap.cc:654] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.084838 7796 raft_consensus.cc:491] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:59.085294 7796 raft_consensus.cc:513] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:57:59.085937 6980 catalog_manager.cc:4852] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet 02b0bf75f58e4a02a8c7def0bff5119f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 02b0bf75f58e4a02a8c7def0bff5119f already in progress: creating tablet
I20250110 20:57:59.087368 7796 leader_election.cc:290] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:59.088470 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:59.088822 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:59.089395 7208 raft_consensus.cc:2463] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:57:59.090528 7287 leader_election.cc:304] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:59.091531 7796 raft_consensus.cc:2798] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250110 20:57:59.091887 7796 raft_consensus.cc:491] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:59.092239 7796 raft_consensus.cc:3054] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:59.103487 7609 tablet_bootstrap.cc:492] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:59.103871 7609 ts_tablet_manager.cc:1397] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.075s user 0.007s sys 0.005s
I20250110 20:57:59.099186 7796 raft_consensus.cc:513] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.106017 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:59.105968 7609 raft_consensus.cc:357] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.106384 7616 ts_tablet_manager.cc:1905] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.106580 7609 raft_consensus.cc:383] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.106566 7208 raft_consensus.cc:3054] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Advancing to term 2
I20250110 20:57:59.106813 7609 raft_consensus.cc:738] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.107393 7609 consensus_queue.cc:260] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.111487 7602 tablet_bootstrap.cc:492] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:59.111388 6991 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_0433916f941242f994c050f80847e23e" } modify_external_catalogs: true
I20250110 20:57:59.117743 7609 ts_tablet_manager.cc:1428] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20250110 20:57:59.118112 7602 tablet_bootstrap.cc:654] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.121626 7609 tablet_bootstrap.cc:492] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:59.122789 6990 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_391e103e657c4fa2914f89bf58d900be" } new_table_name: "default.Table_ba5d6def0428488589bda04057a79fab" modify_external_catalogs: true
I20250110 20:57:59.126044 7802 raft_consensus.cc:491] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:59.126466 7802 raft_consensus.cc:513] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.128226 7802 leader_election.cc:290] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:59.128930 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fb488db4247b3bad2a4905d160fbd" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:59.129113 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fb488db4247b3bad2a4905d160fbd" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:57:59.129549 7352 raft_consensus.cc:2463] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:59.129704 7501 raft_consensus.cc:2463] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:57:59.130482 7145 leader_election.cc:304] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:59.131584 7802 raft_consensus.cc:2798] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:59.131916 7802 raft_consensus.cc:491] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:59.132217 7802 raft_consensus.cc:3054] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.132805 7606 tablet_bootstrap.cc:492] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:59.132750 7208 raft_consensus.cc:2463] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 2.
I20250110 20:57:59.135519 7609 tablet_bootstrap.cc:654] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.135622 7287 leader_election.cc:304] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:59.136530 7762 raft_consensus.cc:2798] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 2 FOLLOWER]: Leader election won for term 2
I20250110 20:57:59.136958 7762 raft_consensus.cc:695] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 2 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:57:59.137722 7762 consensus_queue.cc:237] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.134737 7606 ts_tablet_manager.cc:1397] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.230s user 0.013s sys 0.003s
I20250110 20:57:59.145960 7609 tablet_bootstrap.cc:492] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:59.146368 7609 ts_tablet_manager.cc:1397] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20250110 20:57:59.148738 7609 raft_consensus.cc:357] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.149456 7609 raft_consensus.cc:383] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.149690 7609 raft_consensus.cc:738] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.150641 6994 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 5efa6746553f4b308c923df89b67de32
I20250110 20:57:59.152565 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8632bafec7c408a92328b41ad9e4eef" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:59.152838 6994 catalog_manager.cc:6596] ReplaceTablet: tablet 5efa6746553f4b308c923df89b67de32 deleted and replaced by tablet 9f5d51c407bd4e67ac04c64879a18e5d
I20250110 20:57:59.153967 6991 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_0433916f941242f994c050f80847e23e" } modify_external_catalogs: true
I20250110 20:57:59.154234 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efa6746553f4b308c923df89b67de32 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d51c407bd4e67ac04c64879a18e5d) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:59.154731 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efa6746553f4b308c923df89b67de32 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d51c407bd4e67ac04c64879a18e5d) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:59.155400 7615 tablet_replica.cc:331] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:59.156071 7615 raft_consensus.cc:2238] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.156556 7615 raft_consensus.cc:2267] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.158293 7609 consensus_queue.cc:260] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.159019 7615 ts_tablet_manager.cc:1905] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.157662 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 5efa6746553f4b308c923df89b67de32 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d51c407bd4e67ac04c64879a18e5d) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.160396 7617 tablet_replica.cc:331] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:59.161078 7617 raft_consensus.cc:2238] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:59.162058 7617 raft_consensus.cc:2267] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.163866 7609 ts_tablet_manager.cc:1428] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20250110 20:57:59.164528 7617 ts_tablet_manager.cc:1905] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.165922 7796 leader_election.cc:290] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 2 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:59.168849 7802 raft_consensus.cc:513] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.171452 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fb488db4247b3bad2a4905d160fbd" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:59.172053 7352 raft_consensus.cc:3054] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.178521 7802 leader_election.cc:290] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:57:59.179167 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2fb488db4247b3bad2a4905d160fbd" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:57:59.176838 7606 raft_consensus.cc:357] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:59.177981 7352 raft_consensus.cc:2463] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:59.179726 7501 raft_consensus.cc:3054] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.179829 7606 raft_consensus.cc:383] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.180114 7606 raft_consensus.cc:738] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.180680 7606 consensus_queue.cc:260] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:59.182041 7145 leader_election.cc:304] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:57:59.182585 7602 tablet_bootstrap.cc:492] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:59.182987 7602 ts_tablet_manager.cc:1397] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.072s user 0.009s sys 0.003s
I20250110 20:57:59.189693 7501 raft_consensus.cc:2463] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:57:59.185163 7602 raft_consensus.cc:357] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.193949 7602 raft_consensus.cc:383] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.194270 7602 raft_consensus.cc:738] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.194890 7602 consensus_queue.cc:260] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.198096 6992 catalog_manager.cc:5526] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 1 to 2, leader changed from c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195) to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 2 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:59.207635 6994 master_service.cc:946] ReplaceTablet: received request to replace tablet 669acec9184749f094421b99bea0588f from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:59.214983 7802 raft_consensus.cc:2798] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 20:57:59.224256 6992 catalog_manager.cc:5359] Ignoring report from unknown tablet 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:57:59.232434 7606 ts_tablet_manager.cc:1428] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.086s user 0.003s sys 0.002s
I20250110 20:57:59.234287 7616 ts_tablet_manager.cc:1918] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:59.234668 7616 log.cc:1198] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:57:59.235450 7616 ts_tablet_manager.cc:1939] T 02b0bf75f58e4a02a8c7def0bff5119f P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
W20250110 20:57:59.239215 6992 catalog_manager.cc:5359] Ignoring report from unknown tablet 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:57:59.245555 7606 tablet_bootstrap.cc:492] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:59.252388 7617 ts_tablet_manager.cc:1918] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:59.252803 7617 log.cc:1198] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/5efa6746553f4b308c923df89b67de32
I20250110 20:57:59.254873 7602 ts_tablet_manager.cc:1428] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.072s user 0.006s sys 0.001s
I20250110 20:57:59.257107 7802 raft_consensus.cc:695] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:57:59.257900 7802 consensus_queue.cc:237] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.262866 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 02b0bf75f58e4a02a8c7def0bff5119f (table default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
W20250110 20:57:59.263957 6980 catalog_manager.cc:4839] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 02b0bf75f58e4a02a8c7def0bff5119f because the tablet was not found. No further retry: Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:57:59.264542 7616 tablet_replica.cc:331] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:59.265219 7616 raft_consensus.cc:2238] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.265705 7616 raft_consensus.cc:2267] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.267849 6994 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 669acec9184749f094421b99bea0588f
I20250110 20:57:59.269001 7616 ts_tablet_manager.cc:1905] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.270244 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 669acec9184749f094421b99bea0588f with delete_type TABLET_DATA_DELETED (replaced by tablet 164f26db984343a390f253aba1219f4a) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:59.271526 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 669acec9184749f094421b99bea0588f with delete_type TABLET_DATA_DELETED (replaced by tablet 164f26db984343a390f253aba1219f4a) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:59.273727 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 669acec9184749f094421b99bea0588f with delete_type TABLET_DATA_DELETED (replaced by tablet 164f26db984343a390f253aba1219f4a) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.271760 6994 catalog_manager.cc:6596] ReplaceTablet: tablet 669acec9184749f094421b99bea0588f deleted and replaced by tablet 164f26db984343a390f253aba1219f4a
I20250110 20:57:59.284154 7606 tablet_bootstrap.cc:654] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.289901 7617 ts_tablet_manager.cc:1939] T 5efa6746553f4b308c923df89b67de32 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:59.292807 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:59.293649 7617 tablet_replica.cc:331] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:59.294406 7617 raft_consensus.cc:2238] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.298626 7616 ts_tablet_manager.cc:1918] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:59.298982 7616 log.cc:1198] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/5efa6746553f4b308c923df89b67de32
I20250110 20:57:59.299608 7616 ts_tablet_manager.cc:1939] T 5efa6746553f4b308c923df89b67de32 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:59.301798 7616 tablet_replica.cc:331] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:59.302021 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:59.302433 7616 raft_consensus.cc:2238] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:59.303236 7616 raft_consensus.cc:2267] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.306083 7616 ts_tablet_manager.cc:1905] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.306802 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.311093 6992 catalog_manager.cc:5526] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:57:59.315147 7787 raft_consensus.cc:491] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:59.315548 7787 raft_consensus.cc:513] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.317256 7787 leader_election.cc:290] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:57:59.319799 7616 ts_tablet_manager.cc:1918] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:59.320159 7616 log.cc:1198] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/669acec9184749f094421b99bea0588f
I20250110 20:57:59.320819 7616 ts_tablet_manager.cc:1939] T 669acec9184749f094421b99bea0588f P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:59.321707 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f5d51c407bd4e67ac04c64879a18e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.324846 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.325788 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f5d51c407bd4e67ac04c64879a18e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.325734 7615 ts_tablet_manager.cc:1918] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:59.325838 6985 catalog_manager.cc:2149] created HMS catalog entry for table [id=e3071b63c99e4526910dae97667de29e]
I20250110 20:57:59.326069 7615 log.cc:1198] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/5efa6746553f4b308c923df89b67de32
I20250110 20:57:59.326709 7615 ts_tablet_manager.cc:1939] T 5efa6746553f4b308c923df89b67de32 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:59.329116 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 5efa6746553f4b308c923df89b67de32 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:57:59.329260 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c194735eea7847e1a6abbbdcf34e9579" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:59.329924 7208 raft_consensus.cc:2463] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:59.329039 7615 tablet_replica.cc:331] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:59.331312 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:59.331581 7435 leader_election.cc:304] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:59.332965 7787 raft_consensus.cc:2798] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:59.333094 7615 raft_consensus.cc:2238] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.333338 7787 raft_consensus.cc:491] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:59.319379 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.333935 7787 raft_consensus.cc:3054] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.333990 7615 raft_consensus.cc:2267] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.334518 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f5d51c407bd4e67ac04c64879a18e5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.320502 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c194735eea7847e1a6abbbdcf34e9579" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:59.325237 7606 tablet_bootstrap.cc:492] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:59.341363 7606 ts_tablet_manager.cc:1397] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.103s user 0.009s sys 0.000s
W20250110 20:57:59.341845 7436 leader_election.cc:343] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:59.343611 7606 raft_consensus.cc:357] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.344239 7606 raft_consensus.cc:383] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.344478 7606 raft_consensus.cc:738] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.345013 7606 consensus_queue.cc:260] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.345743 7617 raft_consensus.cc:2267] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.350430 7615 ts_tablet_manager.cc:1905] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.350446 6985 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 16) took 6054 ms. Trace:
I20250110 20:57:59.354800 7617 ts_tablet_manager.cc:1905] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.350744 6985 rpcz_store.cc:276] 0110 20:57:53.296490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:53.299516 (+ 3026us) service_pool.cc:225] Handling call
0110 20:57:58.354972 (+5055456us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:58.363479 (+ 8507us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:59.325799 (+962320us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:57:59.349780 (+ 23981us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:57:59.350127 (+ 347us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:57:59.350381 (+ 254us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:59.327526 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:59.327855 (+ 329us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:59.327877 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:57:59.327901 (+ 24us) tablet.cc:659] Decoding operations
0110 20:57:59.328146 (+ 245us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:59.328178 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:59.328198 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:59.328384 (+ 186us) tablet.cc:698] Row locks acquired
0110 20:57:59.328397 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:57:59.328490 (+ 93us) write_op.cc:270] Start()
0110 20:57:59.328560 (+ 70us) write_op.cc:276] Timestamp: P: 1736542679328432 usec, L: 0
0110 20:57:59.328572 (+ 12us) op_driver.cc:348] REPLICATION: starting
0110 20:57:59.329854 (+ 1282us) log.cc:844] Serialized 432 byte log entry
0110 20:57:59.347623 (+ 17769us) op_driver.cc:464] REPLICATION: finished
0110 20:57:59.348048 (+ 425us) write_op.cc:301] APPLY: starting
0110 20:57:59.348100 (+ 52us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:59.348201 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:59.348293 (+ 92us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:59.348686 (+ 393us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:59.348814 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:59.348854 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:57:59.349028 (+ 174us) log.cc:844] Serialized 28 byte log entry
0110 20:57:59.349166 (+ 138us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:59.349230 (+ 64us) write_op.cc:454] Released schema lock
0110 20:57:59.349259 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:59.349500 (+ 241us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:59.349522 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":537618,"Hive Metastore.run_cpu_time_us":614,"Hive Metastore.run_wall_time_us":423921,"child_traces":[["op",{"apply.queue_time_us":284,"apply.run_cpu_time_us":1376,"apply.run_wall_time_us":1377,"num_ops":2,"prepare.queue_time_us":207,"prepare.run_cpu_time_us":1738,"prepare.run_wall_time_us":2722,"raft.queue_time_us":79,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":18963}]]}
I20250110 20:57:59.359627 7787 raft_consensus.cc:513] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.361960 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet c194735eea7847e1a6abbbdcf34e9579 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:59.362844 7606 ts_tablet_manager.cc:1428] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20250110 20:57:59.363826 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c194735eea7847e1a6abbbdcf34e9579" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:59.364348 7352 raft_consensus.cc:3054] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.365247 7606 tablet_bootstrap.cc:492] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:59.365576 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c194735eea7847e1a6abbbdcf34e9579" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:59.366113 7208 raft_consensus.cc:3054] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.371707 7352 raft_consensus.cc:2463] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.372714 7436 leader_election.cc:304] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:59.373378 7825 raft_consensus.cc:2798] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:59.373787 7825 raft_consensus.cc:695] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:59.374524 7825 consensus_queue.cc:237] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.380772 7602 tablet_bootstrap.cc:492] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:59.381500 7606 tablet_bootstrap.cc:654] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.383764 7749 raft_consensus.cc:491] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:59.384173 7749 raft_consensus.cc:513] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.385684 7602 tablet_bootstrap.cc:654] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.386000 7749 leader_election.cc:290] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:59.386873 6985 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet c194735eea7847e1a6abbbdcf34e9579
I20250110 20:57:59.387107 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e73e60dfdab34bee95fd3bfc2fede7be" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
W20250110 20:57:59.388181 7436 leader_election.cc:343] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:59.388362 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet c194735eea7847e1a6abbbdcf34e9579 with delete_type TABLET_DATA_DELETED (replaced by tablet 53bbd84258e04c76a2d1bf0af353444b) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.389102 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet c194735eea7847e1a6abbbdcf34e9579 with delete_type TABLET_DATA_DELETED (replaced by tablet 53bbd84258e04c76a2d1bf0af353444b) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:59.389714 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e73e60dfdab34bee95fd3bfc2fede7be" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:59.390321 7209 raft_consensus.cc:2463] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:59.391541 6985 catalog_manager.cc:6596] ReplaceTablet: tablet c194735eea7847e1a6abbbdcf34e9579 deleted and replaced by tablet 53bbd84258e04c76a2d1bf0af353444b
I20250110 20:57:59.391381 7435 leader_election.cc:304] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:57:59.392344 7749 raft_consensus.cc:2798] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:59.392644 7749 raft_consensus.cc:491] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:59.392938 7749 raft_consensus.cc:3054] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.398002 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet c194735eea7847e1a6abbbdcf34e9579 with delete_type TABLET_DATA_DELETED (replaced by tablet 53bbd84258e04c76a2d1bf0af353444b) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:59.398617 7616 tablet_replica.cc:331] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:59.400465 7749 raft_consensus.cc:513] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.402976 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e73e60dfdab34bee95fd3bfc2fede7be" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:59.403512 7209 raft_consensus.cc:3054] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.404954 7615 ts_tablet_manager.cc:1918] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:59.405256 7615 log.cc:1198] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/669acec9184749f094421b99bea0588f
I20250110 20:57:59.405903 7615 ts_tablet_manager.cc:1939] T 669acec9184749f094421b99bea0588f P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:59.407958 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:59.409142 7209 raft_consensus.cc:2463] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.410252 7435 leader_election.cc:304] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:59.414427 7844 raft_consensus.cc:2798] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:59.398185 7208 raft_consensus.cc:2463] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.415645 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 164f26db984343a390f253aba1219f4a (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.416558 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164f26db984343a390f253aba1219f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.416908 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 164f26db984343a390f253aba1219f4a (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.417816 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164f26db984343a390f253aba1219f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.427974 7749 leader_election.cc:290] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:59.429298 7615 tablet_replica.cc:331] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:59.430030 7615 raft_consensus.cc:2238] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.430433 7615 raft_consensus.cc:2267] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.432307 7606 tablet_bootstrap.cc:492] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:59.432686 7606 ts_tablet_manager.cc:1397] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.003s
I20250110 20:57:59.434605 7602 tablet_bootstrap.cc:492] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:59.435034 7602 ts_tablet_manager.cc:1397] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.004s
I20250110 20:57:59.435097 7606 raft_consensus.cc:357] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.435724 7606 raft_consensus.cc:383] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.435985 7606 raft_consensus.cc:738] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.436671 7606 consensus_queue.cc:260] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.437433 7602 raft_consensus.cc:357] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.438519 7606 ts_tablet_manager.cc:1428] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20250110 20:57:59.442525 7615 ts_tablet_manager.cc:1905] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.443778 7602 raft_consensus.cc:383] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.444115 7602 raft_consensus.cc:738] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.444718 7602 consensus_queue.cc:260] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.430459 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 164f26db984343a390f253aba1219f4a (DEFAULT_TABLE table=default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.436558 7616 raft_consensus.cc:2238] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:59.451249 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 164f26db984343a390f253aba1219f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.451776 7616 pending_rounds.cc:62] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Trying to abort 1 pending ops.
I20250110 20:57:59.452030 7616 pending_rounds.cc:69] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878814734012416 op_type: NO_OP noop_request { }
I20250110 20:57:59.452370 7616 raft_consensus.cc:2883] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:59.452658 7616 raft_consensus.cc:2267] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.438764 7787 leader_election.cc:290] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:57:59.445446 7844 raft_consensus.cc:695] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:59.446960 7609 tablet_bootstrap.cc:492] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:59.458168 7844 consensus_queue.cc:237] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.460669 7609 tablet_bootstrap.cc:654] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.467613 7616 ts_tablet_manager.cc:1905] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.471376 6992 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
W20250110 20:57:59.473037 6992 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9f5d51c407bd4e67ac04c64879a18e5d consensus state has no leader
I20250110 20:57:59.475870 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e73e60dfdab34bee95fd3bfc2fede7be" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:59.476410 7352 raft_consensus.cc:3054] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.476480 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 6008006d70c44ad7b5f6667e6b5c2d5f (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.477423 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6008006d70c44ad7b5f6667e6b5c2d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.483899 7352 raft_consensus.cc:2463] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.487565 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet f8632bafec7c408a92328b41ad9e4eef from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:59.500950 7049 catalog_manager.cc:2710] Deleting table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2] in response to Hive Metastore notification log event 11
I20250110 20:57:59.513420 7609 tablet_bootstrap.cc:492] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:59.513803 7609 ts_tablet_manager.cc:1397] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.006s
I20250110 20:57:59.518832 7606 tablet_bootstrap.cc:492] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:59.524703 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 6008006d70c44ad7b5f6667e6b5c2d5f (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.526907 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 53bbd84258e04c76a2d1bf0af353444b (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.527797 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53bbd84258e04c76a2d1bf0af353444b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.530372 6994 catalog_manager.cc:5526] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:59.532049 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6008006d70c44ad7b5f6667e6b5c2d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.537575 7602 ts_tablet_manager.cc:1428] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.102s user 0.005s sys 0.000s
I20250110 20:57:59.540431 7602 tablet_bootstrap.cc:492] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:59.545818 6992 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:59.553723 7606 tablet_bootstrap.cc:654] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.559216 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 53bbd84258e04c76a2d1bf0af353444b (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.560170 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53bbd84258e04c76a2d1bf0af353444b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.560480 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 6008006d70c44ad7b5f6667e6b5c2d5f (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.561393 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6008006d70c44ad7b5f6667e6b5c2d5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.561925 7616 ts_tablet_manager.cc:1918] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:59.565624 6985 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet f8632bafec7c408a92328b41ad9e4eef
I20250110 20:57:59.566946 6985 catalog_manager.cc:6596] ReplaceTablet: tablet f8632bafec7c408a92328b41ad9e4eef deleted and replaced by tablet 623a0b5cf6964f8294a6153510054ac0
I20250110 20:57:59.570338 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet f8632bafec7c408a92328b41ad9e4eef with delete_type TABLET_DATA_DELETED (replaced by tablet 623a0b5cf6964f8294a6153510054ac0) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.569145 6992 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_34c642de1a56476a83c2b97f9cdd1287"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:57:59.570344 7609 raft_consensus.cc:357] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.570981 7609 raft_consensus.cc:383] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.571231 7609 raft_consensus.cc:738] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.571763 7609 consensus_queue.cc:260] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.573237 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 53bbd84258e04c76a2d1bf0af353444b (DEFAULT_TABLE table=default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.574155 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53bbd84258e04c76a2d1bf0af353444b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.574294 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet f8632bafec7c408a92328b41ad9e4eef with delete_type TABLET_DATA_DELETED (replaced by tablet 623a0b5cf6964f8294a6153510054ac0) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:59.575729 7616 log.cc:1198] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/c194735eea7847e1a6abbbdcf34e9579
I20250110 20:57:59.576404 7616 ts_tablet_manager.cc:1939] T c194735eea7847e1a6abbbdcf34e9579 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:59.576571 7615 ts_tablet_manager.cc:1918] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:59.576895 7615 log.cc:1198] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/c194735eea7847e1a6abbbdcf34e9579
I20250110 20:57:59.577656 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet f8632bafec7c408a92328b41ad9e4eef with delete_type TABLET_DATA_DELETED (replaced by tablet 623a0b5cf6964f8294a6153510054ac0) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:59.584874 7049 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 1d2fb488db4247b3bad2a4905d160fbd
I20250110 20:57:59.585547 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet c194735eea7847e1a6abbbdcf34e9579 (table default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:57:59.586768 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fb488db4247b3bad2a4905d160fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:59 UTC) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.589972 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fb488db4247b3bad2a4905d160fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:59 UTC) from {username='slave'} at 127.0.0.1:42840
I20250110 20:57:59.578634 7609 ts_tablet_manager.cc:1428] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.065s user 0.005s sys 0.000s
I20250110 20:57:59.587435 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2fb488db4247b3bad2a4905d160fbd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:57:59 UTC) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:59.599165 7609 tablet_bootstrap.cc:492] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:59.597690 7616 tablet_replica.cc:331] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:59.600651 7616 raft_consensus.cc:2238] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:59.601236 7616 pending_rounds.cc:62] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Trying to abort 1 pending ops.
I20250110 20:57:59.601452 7616 pending_rounds.cc:69] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878812981796864 op_type: NO_OP noop_request { }
I20250110 20:57:59.601780 7616 raft_consensus.cc:2883] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:59.602062 7616 raft_consensus.cc:2267] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.604682 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:59.608389 7609 tablet_bootstrap.cc:654] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.609699 7602 tablet_bootstrap.cc:654] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.611191 7616 ts_tablet_manager.cc:1905] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.617033 7615 ts_tablet_manager.cc:1939] T c194735eea7847e1a6abbbdcf34e9579 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:59.628010 7617 ts_tablet_manager.cc:1918] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:57:59.628432 7617 log.cc:1198] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/669acec9184749f094421b99bea0588f
I20250110 20:57:59.629101 7617 ts_tablet_manager.cc:1939] T 669acec9184749f094421b99bea0588f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:59.633628 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 669acec9184749f094421b99bea0588f (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:57:59.635602 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet c194735eea7847e1a6abbbdcf34e9579 (table default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:57:59.636592 7615 tablet_replica.cc:331] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:59.641832 7606 tablet_bootstrap.cc:492] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:59.646035 6994 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:59.650681 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 623a0b5cf6964f8294a6153510054ac0 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.651640 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623a0b5cf6964f8294a6153510054ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.652563 7606 ts_tablet_manager.cc:1397] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.134s user 0.002s sys 0.013s
I20250110 20:57:59.654927 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 623a0b5cf6964f8294a6153510054ac0 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.655820 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623a0b5cf6964f8294a6153510054ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.655864 6994 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:59.658125 7617 tablet_replica.cc:331] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:59.658744 7617 raft_consensus.cc:2238] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.659158 7615 raft_consensus.cc:2238] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.659189 7617 raft_consensus.cc:2267] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.660156 7615 raft_consensus.cc:2267] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.660355 7606 raft_consensus.cc:357] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.660969 7606 raft_consensus.cc:383] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.661206 7606 raft_consensus.cc:738] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.661813 7615 ts_tablet_manager.cc:1905] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.662024 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 623a0b5cf6964f8294a6153510054ac0 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.662837 7602 tablet_bootstrap.cc:492] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:59.662894 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 623a0b5cf6964f8294a6153510054ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.663201 7602 ts_tablet_manager.cc:1397] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.003s
I20250110 20:57:59.664382 7617 ts_tablet_manager.cc:1905] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.665254 7602 raft_consensus.cc:357] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.675056 7602 raft_consensus.cc:383] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.678257 7602 raft_consensus.cc:738] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.682528 6993 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33774 (request call id 13) took 6587 ms. Trace:
I20250110 20:57:59.682845 6993 rpcz_store.cc:276] 0110 20:57:53.094866 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:53.095023 (+ 157us) service_pool.cc:225] Handling call
0110 20:57:58.352856 (+5257833us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:57:59.682463 (+1329607us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":8211,"Hive Metastore.run_cpu_time_us":275,"Hive Metastore.run_wall_time_us":517349}
I20250110 20:57:59.661796 7606 consensus_queue.cc:260] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.690868 7606 ts_tablet_manager.cc:1428] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.037s user 0.004s sys 0.000s
I20250110 20:57:59.691622 7606 tablet_bootstrap.cc:492] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:59.691906 6993 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
W20250110 20:57:59.695572 6994 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:57:59.701112 6994 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:59.711730 7609 tablet_bootstrap.cc:492] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:59.712145 7609 ts_tablet_manager.cc:1397] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.004s
I20250110 20:57:59.713872 7617 ts_tablet_manager.cc:1918] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:59.714200 7617 log.cc:1198] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/c194735eea7847e1a6abbbdcf34e9579
I20250110 20:57:59.678807 7602 consensus_queue.cc:260] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.714748 7617 ts_tablet_manager.cc:1939] T c194735eea7847e1a6abbbdcf34e9579 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:59.722173 7602 ts_tablet_manager.cc:1428] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.059s user 0.006s sys 0.001s
I20250110 20:57:59.727607 6994 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
W20250110 20:57:59.729357 6994 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 164f26db984343a390f253aba1219f4a consensus state has no leader
I20250110 20:57:59.730224 7602 tablet_bootstrap.cc:492] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:57:59.732517 7844 raft_consensus.cc:491] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:59.734526 7844 raft_consensus.cc:513] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.735214 6994 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:59.736439 7844 leader_election.cc:290] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:59.738692 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d51c407bd4e67ac04c64879a18e5d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:59.738688 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d51c407bd4e67ac04c64879a18e5d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:59.739171 7615 ts_tablet_manager.cc:1918] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:59.739284 7208 raft_consensus.cc:2463] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:59.739284 7352 raft_consensus.cc:2463] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:59.739492 7615 log.cc:1198] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/f8632bafec7c408a92328b41ad9e4eef
I20250110 20:57:59.740064 7615 ts_tablet_manager.cc:1939] T f8632bafec7c408a92328b41ad9e4eef P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:59.740309 7436 leader_election.cc:304] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:59.740700 7616 ts_tablet_manager.cc:1918] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:59.741039 7616 log.cc:1198] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/f8632bafec7c408a92328b41ad9e4eef
I20250110 20:57:59.741308 7844 raft_consensus.cc:2798] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:59.741667 7844 raft_consensus.cc:491] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:59.741973 7844 raft_consensus.cc:3054] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.742201 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet f8632bafec7c408a92328b41ad9e4eef (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:59.743010 7615 tablet_replica.cc:331] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:57:59.743258 7602 tablet_bootstrap.cc:654] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.743639 7615 raft_consensus.cc:2238] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:57:59.744246 7615 pending_rounds.cc:62] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:57:59.744450 7615 pending_rounds.cc:69] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878814249291776 op_type: NO_OP noop_request { }
I20250110 20:57:59.744781 7615 raft_consensus.cc:2883] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:59.745079 7615 raft_consensus.cc:2267] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.729727 7609 raft_consensus.cc:357] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.745944 7609 raft_consensus.cc:383] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.746202 7609 raft_consensus.cc:738] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.746344 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet c194735eea7847e1a6abbbdcf34e9579 (table default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:57:59.746850 7609 consensus_queue.cc:260] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.748220 7617 tablet_replica.cc:331] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:59.748148 7844 raft_consensus.cc:513] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.754150 7617 raft_consensus.cc:2238] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 2 LEADER]: Raft consensus shutting down.
I20250110 20:57:59.754561 6994 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:59.754779 7617 pending_rounds.cc:62] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:57:59.755023 7617 pending_rounds.cc:69] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 2 index: 1 } timestamp: 7112878813763485696 op_type: NO_OP noop_request { }
I20250110 20:57:59.756902 7787 raft_consensus.cc:491] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:59.757323 7787 raft_consensus.cc:513] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.759176 7787 leader_election.cc:290] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:59.759887 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f26db984343a390f253aba1219f4a" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:57:59.760464 7208 raft_consensus.cc:2463] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:57:59.760851 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f26db984343a390f253aba1219f4a" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:57:59.761610 7435 leader_election.cc:304] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
W20250110 20:57:59.762318 7436 leader_election.cc:343] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:59.763062 7787 raft_consensus.cc:2798] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:57:59.763381 7787 raft_consensus.cc:491] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:57:59.763669 7787 raft_consensus.cc:3054] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.755415 7617 raft_consensus.cc:2883] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:57:59.764595 7617 raft_consensus.cc:2267] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 2 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.765719 7615 ts_tablet_manager.cc:1905] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.769959 7609 ts_tablet_manager.cc:1428] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.058s user 0.007s sys 0.000s
I20250110 20:57:59.771224 7609 tablet_bootstrap.cc:492] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:59.770447 7787 raft_consensus.cc:513] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.773222 7616 ts_tablet_manager.cc:1939] T f8632bafec7c408a92328b41ad9e4eef P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:59.775470 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet f8632bafec7c408a92328b41ad9e4eef (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:59.776664 6994 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:59.777388 6993 master_service.cc:946] ReplaceTablet: received request to replace tablet 6008006d70c44ad7b5f6667e6b5c2d5f from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:59.778937 7844 leader_election.cc:290] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:59.778805 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d51c407bd4e67ac04c64879a18e5d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:59.779340 7208 raft_consensus.cc:3054] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.779918 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5d51c407bd4e67ac04c64879a18e5d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:57:59.780464 7352 raft_consensus.cc:3054] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.785725 7208 raft_consensus.cc:2463] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.786715 7435 leader_election.cc:304] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:57:59.787231 7352 raft_consensus.cc:2463] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.787678 7606 tablet_bootstrap.cc:654] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.789112 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f26db984343a390f253aba1219f4a" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:57:59.789561 7844 raft_consensus.cc:2798] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:59.789654 7208 raft_consensus.cc:3054] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:57:59.789871 7616 tablet_replica.cc:331] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:57:59.787381 7617 ts_tablet_manager.cc:1905] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.790622 7844 raft_consensus.cc:695] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:59.792722 7787 leader_election.cc:290] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:57:59.792454 7844 consensus_queue.cc:237] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.791738 7616 raft_consensus.cc:2238] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.794802 7616 raft_consensus.cc:2267] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.793502 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f26db984343a390f253aba1219f4a" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
W20250110 20:57:59.796075 7436 leader_election.cc:343] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:57:59.797967 7609 tablet_bootstrap.cc:654] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.798513 7208 raft_consensus.cc:2463] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:57:59.799894 7602 tablet_bootstrap.cc:492] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:59.800390 7602 ts_tablet_manager.cc:1397] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.005s
I20250110 20:57:59.802490 7602 raft_consensus.cc:357] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.799479 7435 leader_election.cc:304] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:57:59.803479 7787 raft_consensus.cc:2798] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:57:59.803947 7787 raft_consensus.cc:695] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:57:59.804980 7787 consensus_queue.cc:237] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.806813 7617 ts_tablet_manager.cc:1918] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20250110 20:57:59.807179 7617 log.cc:1198] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f8632bafec7c408a92328b41ad9e4eef
I20250110 20:57:59.807816 7617 ts_tablet_manager.cc:1939] T f8632bafec7c408a92328b41ad9e4eef P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:57:59.809517 7602 raft_consensus.cc:383] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.807878 7616 ts_tablet_manager.cc:1905] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.810060 7602 raft_consensus.cc:738] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.813771 7602 consensus_queue.cc:260] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.832309 7602 ts_tablet_manager.cc:1428] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20250110 20:57:59.831096 6985 catalog_manager.cc:5526] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:59.835338 7602 tablet_bootstrap.cc:492] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
W20250110 20:57:59.837776 6979 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9f5d51c407bd4e67ac04c64879a18e5d consensus state has no leader
I20250110 20:57:59.834197 6985 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:57:59.838196 7606 tablet_bootstrap.cc:492] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:57:59.838605 7606 ts_tablet_manager.cc:1397] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.147s user 0.008s sys 0.001s
I20250110 20:57:59.845001 7606 raft_consensus.cc:357] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.853818 7606 raft_consensus.cc:383] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.856747 7615 ts_tablet_manager.cc:1918] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:57:59.857112 7615 log.cc:1198] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/1d2fb488db4247b3bad2a4905d160fbd
I20250110 20:57:59.857769 7615 ts_tablet_manager.cc:1939] T 1d2fb488db4247b3bad2a4905d160fbd P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:57:59.860178 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 1d2fb488db4247b3bad2a4905d160fbd (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:57:59.861510 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet f8632bafec7c408a92328b41ad9e4eef (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:57:59.863847 7609 tablet_bootstrap.cc:492] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:59.861789 7617 tablet_replica.cc:331] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:57:59.866571 7617 raft_consensus.cc:2238] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:57:59.866999 7617 raft_consensus.cc:2267] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:57:59.868676 7617 ts_tablet_manager.cc:1905] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:57:59.869136 7609 ts_tablet_manager.cc:1397] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.004s
I20250110 20:57:59.871275 7602 tablet_bootstrap.cc:654] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.871217 6993 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:57:59.872560 6993 catalog_manager.cc:6596] ReplaceTablet: tablet 6008006d70c44ad7b5f6667e6b5c2d5f deleted and replaced by tablet ad0d5f590ed34563be385fe98a149a3f
I20250110 20:57:59.854096 7606 raft_consensus.cc:738] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.873162 6993 master_service.cc:946] ReplaceTablet: received request to replace tablet 623a0b5cf6964f8294a6153510054ac0 from {username='slave'} at 127.0.0.1:33774
I20250110 20:57:59.873695 7606 consensus_queue.cc:260] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.875589 7609 raft_consensus.cc:357] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.876201 7609 raft_consensus.cc:383] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.876437 7609 raft_consensus.cc:738] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.877027 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 6008006d70c44ad7b5f6667e6b5c2d5f with delete_type TABLET_DATA_DELETED (replaced by tablet ad0d5f590ed34563be385fe98a149a3f) from {username='slave'} at 127.0.0.1:43396
I20250110 20:57:59.879146 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 6008006d70c44ad7b5f6667e6b5c2d5f with delete_type TABLET_DATA_DELETED (replaced by tablet ad0d5f590ed34563be385fe98a149a3f) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.882402 7616 ts_tablet_manager.cc:1918] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:57:59.882732 7616 log.cc:1198] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/1d2fb488db4247b3bad2a4905d160fbd
I20250110 20:57:59.883319 7616 ts_tablet_manager.cc:1939] T 1d2fb488db4247b3bad2a4905d160fbd P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:57:59.886955 7606 ts_tablet_manager.cc:1428] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20250110 20:57:59.888367 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 1d2fb488db4247b3bad2a4905d160fbd (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
W20250110 20:57:59.888993 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 6008006d70c44ad7b5f6667e6b5c2d5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6008006d70c44ad7b5f6667e6b5c2d5f already in progress: creating tablet
I20250110 20:57:59.893680 6994 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_7a871347a918413297f7da917134a92e" } modify_external_catalogs: true
I20250110 20:57:59.894285 6994 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_7a871347a918413297f7da917134a92e" } modify_external_catalogs: true
I20250110 20:57:59.896315 7609 consensus_queue.cc:260] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.898240 7606 tablet_bootstrap.cc:492] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:57:59.875074 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 6008006d70c44ad7b5f6667e6b5c2d5f with delete_type TABLET_DATA_DELETED (replaced by tablet ad0d5f590ed34563be385fe98a149a3f) from {username='slave'} at 127.0.0.1:42840
W20250110 20:57:59.901429 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet 6008006d70c44ad7b5f6667e6b5c2d5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6008006d70c44ad7b5f6667e6b5c2d5f already in progress: creating tablet
W20250110 20:57:59.911109 6985 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:57:59.913792 7609 ts_tablet_manager.cc:1428] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.040s user 0.004s sys 0.002s
I20250110 20:57:59.914677 7609 tablet_bootstrap.cc:492] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:57:59.923828 7609 tablet_bootstrap.cc:654] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.928457 6993 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 623a0b5cf6964f8294a6153510054ac0
I20250110 20:57:59.930001 6993 catalog_manager.cc:6596] ReplaceTablet: tablet 623a0b5cf6964f8294a6153510054ac0 deleted and replaced by tablet eff2ce38b9864782b1a1a27d05c9d777
I20250110 20:57:59.933930 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 623a0b5cf6964f8294a6153510054ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet eff2ce38b9864782b1a1a27d05c9d777) from {username='slave'} at 127.0.0.1:42840
W20250110 20:57:59.935631 6979 catalog_manager.cc:4852] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet 623a0b5cf6964f8294a6153510054ac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 623a0b5cf6964f8294a6153510054ac0 already in progress: creating tablet
I20250110 20:57:59.938985 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 623a0b5cf6964f8294a6153510054ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet eff2ce38b9864782b1a1a27d05c9d777) from {username='slave'} at 127.0.0.1:40394
I20250110 20:57:59.944299 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 623a0b5cf6964f8294a6153510054ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet eff2ce38b9864782b1a1a27d05c9d777) from {username='slave'} at 127.0.0.1:43396
W20250110 20:57:59.945936 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 623a0b5cf6964f8294a6153510054ac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 623a0b5cf6964f8294a6153510054ac0 already in progress: creating tablet
I20250110 20:57:59.947856 6987 catalog_manager.cc:2149] created HMS catalog entry for table [id=bbced07fb45947d4aaa1e1e4a08756da]
I20250110 20:57:59.953835 7602 tablet_bootstrap.cc:492] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:59.955547 7602 ts_tablet_manager.cc:1397] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.120s user 0.013s sys 0.014s
I20250110 20:57:59.957798 7602 raft_consensus.cc:357] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.958417 7602 raft_consensus.cc:383] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.958636 7602 raft_consensus.cc:738] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.960961 7602 consensus_queue.cc:260] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.960601 6985 catalog_manager.cc:5526] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:57:59.961938 6985 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:57:59.962901 7796 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:57:59.963356 7796 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:57:59.964430 7602 ts_tablet_manager.cc:1428] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250110 20:57:59.965222 7602 tablet_bootstrap.cc:492] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
W20250110 20:57:59.970009 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:57:59.970642 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:57:59.970981 7288 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:57:59.965219 7796 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:57:59.971983 7796 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:57:59.971935 7481 tablet_service.cc:1467] Processing CreateTablet for tablet ad0d5f590ed34563be385fe98a149a3f (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.972903 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0d5f590ed34563be385fe98a149a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.976295 7602 tablet_bootstrap.cc:654] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.976250 7190 tablet_service.cc:1467] Processing CreateTablet for tablet ad0d5f590ed34563be385fe98a149a3f (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:57:59.977193 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0d5f590ed34563be385fe98a149a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:57:59.980692 7606 tablet_bootstrap.cc:654] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:57:59.988328 7602 tablet_bootstrap.cc:492] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:57:59.988695 7602 ts_tablet_manager.cc:1397] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.002s
I20250110 20:57:59.995651 7602 raft_consensus.cc:357] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:57:59.996263 7602 raft_consensus.cc:383] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:57:59.996484 7602 raft_consensus.cc:738] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:57:59.997812 7609 tablet_bootstrap.cc:492] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:57:59.998198 7609 ts_tablet_manager.cc:1397] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.003s
I20250110 20:57:59.998509 7602 consensus_queue.cc:260] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.005728 6987 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 30) took 5037 ms. Trace:
I20250110 20:58:00.007206 7602 ts_tablet_manager.cc:1428] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20250110 20:58:00.008003 7602 tablet_bootstrap.cc:492] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:00.006038 6987 rpcz_store.cc:276] 0110 20:57:54.968322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:54.968461 (+ 139us) service_pool.cc:225] Handling call
0110 20:57:58.911414 (+3942953us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:58.913469 (+ 2055us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:57:59.947816 (+1034347us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:00.000861 (+ 53045us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:00.005351 (+ 4490us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:00.005665 (+ 314us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:57:59.949585 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:57:59.949892 (+ 307us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:57:59.949913 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:57:59.949925 (+ 12us) tablet.cc:659] Decoding operations
0110 20:57:59.950162 (+ 237us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:57:59.950194 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0110 20:57:59.950214 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:57:59.950374 (+ 160us) tablet.cc:698] Row locks acquired
0110 20:57:59.950386 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:57:59.950478 (+ 92us) write_op.cc:270] Start()
0110 20:57:59.950551 (+ 73us) write_op.cc:276] Timestamp: P: 1736542679950411 usec, L: 0
0110 20:57:59.950565 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:57:59.950962 (+ 397us) log.cc:844] Serialized 432 byte log entry
0110 20:57:59.996709 (+ 45747us) op_driver.cc:464] REPLICATION: finished
0110 20:57:59.997266 (+ 557us) write_op.cc:301] APPLY: starting
0110 20:57:59.997317 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0110 20:57:59.997430 (+ 113us) tablet.cc:1367] finished BulkCheckPresence
0110 20:57:59.997514 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:57:59.997852 (+ 338us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:57:59.997979 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:57:59.998017 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:57:59.998209 (+ 192us) log.cc:844] Serialized 28 byte log entry
0110 20:57:59.998345 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:59.998409 (+ 64us) write_op.cc:454] Released schema lock
0110 20:57:59.998437 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:57:59.998639 (+ 202us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:57:59.998661 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":576460,"Hive Metastore.run_cpu_time_us":547,"Hive Metastore.run_wall_time_us":456784,"child_traces":[["op",{"apply.queue_time_us":426,"apply.run_cpu_time_us":1340,"apply.run_wall_time_us":1340,"num_ops":2,"prepare.queue_time_us":262,"prepare.run_cpu_time_us":1603,"prepare.run_wall_time_us":1603,"raft.queue_time_us":158,"raft.run_cpu_time_us":15,"raft.run_wall_time_us":15,"replication_time_us":46046}]]}
I20250110 20:58:00.016753 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 6008006d70c44ad7b5f6667e6b5c2d5f with delete_type TABLET_DATA_DELETED (replaced by tablet ad0d5f590ed34563be385fe98a149a3f) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:00.018088 7615 tablet_replica.cc:331] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:00.018731 7615 raft_consensus.cc:2238] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.019129 7615 raft_consensus.cc:2267] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.020241 7602 tablet_bootstrap.cc:654] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.021090 7615 ts_tablet_manager.cc:1905] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.030032 7617 ts_tablet_manager.cc:1918] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.030467 7617 log.cc:1198] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/1d2fb488db4247b3bad2a4905d160fbd
W20250110 20:58:00.028002 7288 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:45717, user_credentials={real_user=slave}} blocked reactor thread for 57494.4us
I20250110 20:58:00.031158 7602 tablet_bootstrap.cc:492] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:00.031517 7602 ts_tablet_manager.cc:1397] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.005s
I20250110 20:58:00.032363 7332 tablet_service.cc:1467] Processing CreateTablet for tablet ad0d5f590ed34563be385fe98a149a3f (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.033262 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0d5f590ed34563be385fe98a149a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.033370 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 623a0b5cf6964f8294a6153510054ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet eff2ce38b9864782b1a1a27d05c9d777) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:00.033615 7602 raft_consensus.cc:357] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:00.037384 7602 raft_consensus.cc:383] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.037638 7602 raft_consensus.cc:738] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.005595 7609 raft_consensus.cc:357] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.038447 7609 raft_consensus.cc:383] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.039158 7602 consensus_queue.cc:260] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:00.042291 7602 ts_tablet_manager.cc:1428] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20250110 20:58:00.055488 7615 ts_tablet_manager.cc:1918] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.055828 7615 log.cc:1198] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:00.056406 7615 ts_tablet_manager.cc:1939] T 6008006d70c44ad7b5f6667e6b5c2d5f P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:00.060956 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad3b006f8364950bcf60d0b0888b819 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.062103 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad3b006f8364950bcf60d0b0888b819 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.062597 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad3b006f8364950bcf60d0b0888b819. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.062983 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad3b006f8364950bcf60d0b0888b819. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.070945 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad3b006f8364950bcf60d0b0888b819 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.071924 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ad3b006f8364950bcf60d0b0888b819. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:58:00.075867 7436 consensus_peers.cc:487] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:58:00.038717 7609 raft_consensus.cc:738] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.076886 7609 consensus_queue.cc:260] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.080857 7208 raft_consensus.cc:1270] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer c3b63d95ca734d8a9803b96bfd94f307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:00.081655 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 6008006d70c44ad7b5f6667e6b5c2d5f (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:00.083284 7615 tablet_replica.cc:331] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:00.084081 7615 raft_consensus.cc:2238] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.084486 7615 raft_consensus.cc:2267] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.086907 7802 raft_consensus.cc:491] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:00.087297 7802 raft_consensus.cc:513] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:00.088626 7615 ts_tablet_manager.cc:1905] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.089023 7802 leader_election.cc:290] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:00.089605 6993 master_service.cc:946] ReplaceTablet: received request to replace tablet 623a0b5cf6964f8294a6153510054ac0 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:00.091866 7844 consensus_queue.cc:1035] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:00.094054 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad0d5f590ed34563be385fe98a149a3f" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
W20250110 20:58:00.095120 7145 leader_election.cc:343] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:58:00.095657 7145 leader_election.cc:343] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: ad0d5f590ed34563be385fe98a149a3f
I20250110 20:58:00.096015 7145 leader_election.cc:304] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:00.097859 7617 ts_tablet_manager.cc:1939] T 1d2fb488db4247b3bad2a4905d160fbd P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:00.099839 7609 ts_tablet_manager.cc:1428] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.101s user 0.006s sys 0.000s
I20250110 20:58:00.100121 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 1d2fb488db4247b3bad2a4905d160fbd (table default.table_52bd7ec724394352a929ba64cab155d2 [id=6526e5cacb274d1097e6d2e6e4f973c2]) successfully deleted
I20250110 20:58:00.100999 7617 tablet_replica.cc:331] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:00.102182 7802 raft_consensus.cc:2743] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:00.103792 7189 tablet_service.cc:1467] Processing CreateTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.104097 7617 raft_consensus.cc:2238] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.104490 7617 raft_consensus.cc:2267] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.104681 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff2ce38b9864782b1a1a27d05c9d777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.108980 7609 tablet_bootstrap.cc:492] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:00.115005 7606 tablet_bootstrap.cc:492] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:00.115437 7606 ts_tablet_manager.cc:1397] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.217s user 0.005s sys 0.004s
I20250110 20:58:00.117139 7615 ts_tablet_manager.cc:1918] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.117498 7615 log.cc:1198] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:00.118094 7615 ts_tablet_manager.cc:1939] T 623a0b5cf6964f8294a6153510054ac0 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:00.117939 7606 raft_consensus.cc:357] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.118567 7606 raft_consensus.cc:383] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.118804 7606 raft_consensus.cc:738] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.119784 7602 tablet_bootstrap.cc:492] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:00.119423 7606 consensus_queue.cc:260] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.124734 7602 tablet_bootstrap.cc:654] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.143949 7602 tablet_bootstrap.cc:492] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:00.145350 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 623a0b5cf6964f8294a6153510054ac0 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:00.149632 6993 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:58:00.150631 7606 ts_tablet_manager.cc:1428] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20250110 20:58:00.158993 7602 ts_tablet_manager.cc:1397] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.004s
I20250110 20:58:00.161935 7332 tablet_service.cc:1467] Processing CreateTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.163403 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff2ce38b9864782b1a1a27d05c9d777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.161692 7602 raft_consensus.cc:357] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.164523 7602 raft_consensus.cc:383] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.166299 6993 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:58:00.169692 7602 raft_consensus.cc:738] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.170285 7602 consensus_queue.cc:260] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.177122 7352 raft_consensus.cc:1270] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Refusing update from remote peer c3b63d95ca734d8a9803b96bfd94f307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:00.178828 7602 ts_tablet_manager.cc:1428] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.019s user 0.004s sys 0.000s
I20250110 20:58:00.181573 7602 tablet_bootstrap.cc:492] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:00.181828 7209 raft_consensus.cc:1270] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer c3b63d95ca734d8a9803b96bfd94f307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:00.137238 7844 raft_consensus.cc:491] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:00.187577 7844 raft_consensus.cc:513] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.190164 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bbd84258e04c76a2d1bf0af353444b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:00.190721 7209 raft_consensus.cc:2463] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:00.191936 7435 leader_election.cc:304] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:00.194416 6985 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:58:00.171128 7481 tablet_service.cc:1467] Processing CreateTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.196812 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eff2ce38b9864782b1a1a27d05c9d777. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.180351 7787 consensus_queue.cc:1035] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20250110 20:58:00.200461 7606 tablet_bootstrap.cc:492] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:00.201112 7787 raft_consensus.cc:2798] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:00.201438 7787 raft_consensus.cc:491] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:00.201738 7787 raft_consensus.cc:3054] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:00.206929 7749 consensus_queue.cc:1035] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:00.211210 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet 9f5d51c407bd4e67ac04c64879a18e5d from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:00.218295 7787 raft_consensus.cc:513] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.221441 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bbd84258e04c76a2d1bf0af353444b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:00.186887 7617 ts_tablet_manager.cc:1905] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:58:00.223659 7436 leader_election.cc:343] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:00.159605 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 623a0b5cf6964f8294a6153510054ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet eff2ce38b9864782b1a1a27d05c9d777) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:00.225189 7616 tablet_replica.cc:331] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:00.225883 7616 raft_consensus.cc:2238] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.226397 7616 raft_consensus.cc:2267] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.227820 7787 leader_election.cc:290] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:00.228396 7616 ts_tablet_manager.cc:1905] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.234607 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bbd84258e04c76a2d1bf0af353444b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:00.235168 7210 raft_consensus.cc:3054] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:00.240561 6993 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_e760ae0763c04467a8fea38edff37362"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:00.241766 7210 raft_consensus.cc:2463] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:00.244194 7606 tablet_bootstrap.cc:654] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.247375 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bbd84258e04c76a2d1bf0af353444b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
W20250110 20:58:00.248783 7436 leader_election.cc:343] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:00.249792 7844 leader_election.cc:290] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:00.250036 7435 leader_election.cc:304] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:00.251122 7874 raft_consensus.cc:2798] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:00.251587 7874 raft_consensus.cc:695] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:58:00.252625 7602 tablet_bootstrap.cc:654] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.252318 7874 consensus_queue.cc:237] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.255640 7609 tablet_bootstrap.cc:700] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307: Time spent opening tablet: real 0.119s user 0.001s sys 0.000s
I20250110 20:58:00.256192 7609 tablet_bootstrap.cc:654] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.260190 7616 ts_tablet_manager.cc:1918] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.260514 7616 log.cc:1198] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:00.261111 7616 ts_tablet_manager.cc:1939] T 623a0b5cf6964f8294a6153510054ac0 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:00.263473 7880 tablet.cc:1721] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:00.283437 7871 mvcc.cc:204] Tried to move back new op lower bound from 7112878817572392960 to 7112878816489463808. Current Snapshot: MvccSnapshot[applied={T|T < 7112878817572392960}]
I20250110 20:58:00.287235 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 623a0b5cf6964f8294a6153510054ac0 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:00.290040 7880 tablet.cc:1721] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:00.310936 7787 consensus_queue.cc:1035] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:00.321300 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 623a0b5cf6964f8294a6153510054ac0 with delete_type TABLET_DATA_DELETED (replaced by tablet eff2ce38b9864782b1a1a27d05c9d777) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:00.322379 7787 consensus_queue.cc:1035] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:00.323410 7886 tablet.cc:1721] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:00.328261 6985 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 9f5d51c407bd4e67ac04c64879a18e5d
I20250110 20:58:00.329069 7606 tablet_bootstrap.cc:492] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:00.329495 7606 ts_tablet_manager.cc:1397] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.129s user 0.010s sys 0.000s
I20250110 20:58:00.330886 6985 catalog_manager.cc:6596] ReplaceTablet: tablet 9f5d51c407bd4e67ac04c64879a18e5d deleted and replaced by tablet ac84d3932d5e421fa0b641419eeebe6a
I20250110 20:58:00.331557 7606 raft_consensus.cc:357] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.332187 7606 raft_consensus.cc:383] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.332409 7606 raft_consensus.cc:738] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.332793 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d with delete_type TABLET_DATA_DELETED (replaced by tablet ac84d3932d5e421fa0b641419eeebe6a) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:00.333467 7615 tablet_replica.cc:331] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:00.334110 7615 raft_consensus.cc:2238] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.334553 7615 raft_consensus.cc:2267] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.342643 6985 catalog_manager.cc:5526] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
W20250110 20:58:00.347539 6980 catalog_manager.cc:4839] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 623a0b5cf6964f8294a6153510054ac0 because the tablet was not found. No further retry: Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:00.346760 7882 tablet.cc:1721] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:00.348742 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d with delete_type TABLET_DATA_DELETED (replaced by tablet ac84d3932d5e421fa0b641419eeebe6a) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:00.349318 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d with delete_type TABLET_DATA_DELETED (replaced by tablet ac84d3932d5e421fa0b641419eeebe6a) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:00.349025 7606 consensus_queue.cc:260] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.349886 7863 raft_consensus.cc:491] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:00.350041 7616 tablet_replica.cc:331] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:00.350273 7863 raft_consensus.cc:513] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.351668 7617 ts_tablet_manager.cc:1918] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.303623 7885 tablet.cc:1721] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:00.356781 7617 log.cc:1198] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:00.356750 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bbd84258e04c76a2d1bf0af353444b" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:00.357502 7208 raft_consensus.cc:2388] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ef4ce3d44cc486fb0ee0ad85127011b in current term 1: Already voted for candidate c3b63d95ca734d8a9803b96bfd94f307 in this term.
I20250110 20:58:00.361649 7617 ts_tablet_manager.cc:1939] T 6008006d70c44ad7b5f6667e6b5c2d5f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:00.365170 7863 leader_election.cc:290] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:00.368513 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bbd84258e04c76a2d1bf0af353444b" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:00.369859 7288 leader_election.cc:304] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:00.374105 7609 tablet_bootstrap.cc:492] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:00.374583 7609 ts_tablet_manager.cc:1397] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.266s user 0.009s sys 0.000s
I20250110 20:58:00.376660 7796 raft_consensus.cc:3054] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:00.378508 7606 ts_tablet_manager.cc:1428] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20250110 20:58:00.379375 7606 tablet_bootstrap.cc:492] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:00.383129 7796 raft_consensus.cc:2743] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:00.384294 7606 tablet_bootstrap.cc:654] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.396597 6989 catalog_manager.cc:2149] created HMS catalog entry for table [id=666e7a40ae294813a1bba6474602377b]
I20250110 20:58:00.398221 7617 tablet_replica.cc:331] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:00.398934 7617 raft_consensus.cc:2238] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.399379 7617 raft_consensus.cc:2267] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.406617 7602 tablet_bootstrap.cc:492] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:00.407009 7602 ts_tablet_manager.cc:1397] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.228s user 0.015s sys 0.016s
I20250110 20:58:00.409071 7602 raft_consensus.cc:357] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.409786 7602 raft_consensus.cc:383] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.410036 7602 raft_consensus.cc:738] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.410679 7602 consensus_queue.cc:260] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.416201 7606 tablet_bootstrap.cc:492] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:00.416587 7606 ts_tablet_manager.cc:1397] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.008s
I20250110 20:58:00.418130 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 6008006d70c44ad7b5f6667e6b5c2d5f (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:00.418812 7606 raft_consensus.cc:357] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.419454 7606 raft_consensus.cc:383] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.419689 7606 raft_consensus.cc:738] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.420213 7606 consensus_queue.cc:260] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.425817 7602 ts_tablet_manager.cc:1428] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20250110 20:58:00.420382 7616 raft_consensus.cc:2238] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:00.430974 7606 ts_tablet_manager.cc:1428] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20250110 20:58:00.430964 7616 raft_consensus.cc:2267] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.438709 7606 tablet_bootstrap.cc:492] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:00.439904 7609 raft_consensus.cc:357] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.440622 7609 raft_consensus.cc:383] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.440979 7609 raft_consensus.cc:738] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.449985 7606 tablet_bootstrap.cc:654] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:00.450613 6980 catalog_manager.cc:4852] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet 623a0b5cf6964f8294a6153510054ac0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 623a0b5cf6964f8294a6153510054ac0 already in progress: creating tablet
I20250110 20:58:00.452302 7617 ts_tablet_manager.cc:1905] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.461681 6985 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:58:00.465907 6987 master_service.cc:946] ReplaceTablet: received request to replace tablet eff2ce38b9864782b1a1a27d05c9d777 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:00.482707 7615 ts_tablet_manager.cc:1905] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:58:00.490646 6980 consensus_peers.cc:487] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 -> Peer f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203): Couldn't send request to peer f9bb61bdf30044a1b482d6a66e29435f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20250110 20:58:00.494679 7616 ts_tablet_manager.cc:1905] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.498162 7796 raft_consensus.cc:491] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:00.498571 7796 raft_consensus.cc:513] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.500391 7796 leader_election.cc:290] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:00.501443 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f26db984343a390f253aba1219f4a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:00.512881 7615 ts_tablet_manager.cc:1918] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:00.513211 7615 log.cc:1198] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/9f5d51c407bd4e67ac04c64879a18e5d
I20250110 20:58:00.513584 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "164f26db984343a390f253aba1219f4a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:00.515045 7287 leader_election.cc:304] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:00.516109 7796 raft_consensus.cc:2743] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:00.516924 7609 consensus_queue.cc:260] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.518234 7606 tablet_bootstrap.cc:492] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:00.518605 7606 ts_tablet_manager.cc:1397] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.080s user 0.014s sys 0.009s
I20250110 20:58:00.521178 7606 raft_consensus.cc:357] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.521807 7606 raft_consensus.cc:383] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.522050 7606 raft_consensus.cc:738] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.522590 7606 consensus_queue.cc:260] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.528188 7606 ts_tablet_manager.cc:1428] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20250110 20:58:00.531411 6989 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 57) took 2226 ms. Trace:
I20250110 20:58:00.531646 7606 tablet_bootstrap.cc:492] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:00.531701 6989 rpcz_store.cc:276] 0110 20:57:58.304925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:58.305058 (+ 133us) service_pool.cc:225] Handling call
0110 20:57:58.913310 (+608252us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:58.914856 (+ 1546us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:58:00.396555 (+1481699us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:00.530828 (+134273us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:00.531110 (+ 282us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:00.531354 (+ 244us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:58:00.398195 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:58:00.398528 (+ 333us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:58:00.398553 (+ 25us) write_op.cc:435] Acquired schema lock
0110 20:58:00.398565 (+ 12us) tablet.cc:659] Decoding operations
0110 20:58:00.398806 (+ 241us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:58:00.398841 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0110 20:58:00.398862 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:58:00.399051 (+ 189us) tablet.cc:698] Row locks acquired
0110 20:58:00.399063 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:58:00.399176 (+ 113us) write_op.cc:270] Start()
0110 20:58:00.399260 (+ 84us) write_op.cc:276] Timestamp: P: 1736542680399112 usec, L: 0
0110 20:58:00.399275 (+ 15us) op_driver.cc:348] REPLICATION: starting
0110 20:58:00.399736 (+ 461us) log.cc:844] Serialized 432 byte log entry
0110 20:58:00.528351 (+128615us) op_driver.cc:464] REPLICATION: finished
0110 20:58:00.528733 (+ 382us) write_op.cc:301] APPLY: starting
0110 20:58:00.528783 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0110 20:58:00.528884 (+ 101us) tablet.cc:1367] finished BulkCheckPresence
0110 20:58:00.528973 (+ 89us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:58:00.529323 (+ 350us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:58:00.529475 (+ 152us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:58:00.529514 (+ 39us) write_op.cc:312] APPLY: finished
0110 20:58:00.529681 (+ 167us) log.cc:844] Serialized 28 byte log entry
0110 20:58:00.529820 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:00.529886 (+ 66us) write_op.cc:454] Released schema lock
0110 20:58:00.529915 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:58:00.530102 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:00.530136 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1032034,"Hive Metastore.run_cpu_time_us":554,"Hive Metastore.run_wall_time_us":448634,"child_traces":[["op",{"apply.queue_time_us":243,"apply.run_cpu_time_us":1347,"apply.run_wall_time_us":1347,"num_ops":2,"prepare.queue_time_us":123,"prepare.run_cpu_time_us":1794,"prepare.run_wall_time_us":1795,"raft.queue_time_us":3009,"raft.run_cpu_time_us":14,"raft.run_wall_time_us":14,"replication_time_us":128987}]]}
W20250110 20:58:00.546062 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:00.547596 7609 ts_tablet_manager.cc:1428] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.110s user 0.006s sys 0.000s
I20250110 20:58:00.513886 7615 ts_tablet_manager.cc:1939] T 9f5d51c407bd4e67ac04c64879a18e5d P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:00.552573 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:58:00.556191 7609 tablet_bootstrap.cc:492] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:00.537395 7617 ts_tablet_manager.cc:1918] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:00.561987 7617 log.cc:1198] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/9f5d51c407bd4e67ac04c64879a18e5d
I20250110 20:58:00.562682 7617 ts_tablet_manager.cc:1939] T 9f5d51c407bd4e67ac04c64879a18e5d P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:00.567694 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 with delete_type TABLET_DATA_DELETED (replaced by tablet f25437547b454eafbcf56e2b6c3b348a) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:00.569573 7616 ts_tablet_manager.cc:1918] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:00.569913 7616 log.cc:1198] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/9f5d51c407bd4e67ac04c64879a18e5d
I20250110 20:58:00.571477 6987 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet eff2ce38b9864782b1a1a27d05c9d777
I20250110 20:58:00.573732 6987 catalog_manager.cc:6596] ReplaceTablet: tablet eff2ce38b9864782b1a1a27d05c9d777 deleted and replaced by tablet f25437547b454eafbcf56e2b6c3b348a
W20250110 20:58:00.576623 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:00.576793 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 with delete_type TABLET_DATA_DELETED (replaced by tablet f25437547b454eafbcf56e2b6c3b348a) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:00.576112 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 with delete_type TABLET_DATA_DELETED (replaced by tablet f25437547b454eafbcf56e2b6c3b348a) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:00.579507 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:58:00.570549 7616 ts_tablet_manager.cc:1939] T 9f5d51c407bd4e67ac04c64879a18e5d P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:00.585068 6986 catalog_manager.cc:2149] created HMS catalog entry for table [id=b0a900b933d349bc8b9b5fc9ce31decb]
I20250110 20:58:00.580068 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
W20250110 20:58:00.588378 6980 catalog_manager.cc:4852] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet eff2ce38b9864782b1a1a27d05c9d777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff2ce38b9864782b1a1a27d05c9d777 already in progress: creating tablet
I20250110 20:58:00.591301 7189 tablet_service.cc:1467] Processing CreateTablet for tablet ac84d3932d5e421fa0b641419eeebe6a (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.592273 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac84d3932d5e421fa0b641419eeebe6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.576318 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 with delete_type TABLET_DATA_DELETED (replaced by tablet f25437547b454eafbcf56e2b6c3b348a) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:00.608076 7332 tablet_service.cc:1467] Processing CreateTablet for tablet ac84d3932d5e421fa0b641419eeebe6a (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.608989 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac84d3932d5e421fa0b641419eeebe6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.609483 6986 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 54) took 2446 ms. Trace:
I20250110 20:58:00.611104 7481 tablet_service.cc:1467] Processing CreateTablet for tablet ac84d3932d5e421fa0b641419eeebe6a (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.612011 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac84d3932d5e421fa0b641419eeebe6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.609798 6986 rpcz_store.cc:276] 0110 20:57:58.162642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:58.162775 (+ 133us) service_pool.cc:225] Handling call
0110 20:57:58.915311 (+752536us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:57:58.916874 (+ 1563us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:58:00.585027 (+1668153us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:00.608836 (+ 23809us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:00.609144 (+ 308us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:00.609423 (+ 279us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:58:00.587273 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:58:00.587578 (+ 305us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:58:00.587597 (+ 19us) write_op.cc:435] Acquired schema lock
0110 20:58:00.587609 (+ 12us) tablet.cc:659] Decoding operations
0110 20:58:00.587840 (+ 231us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:58:00.587881 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0110 20:58:00.587902 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:58:00.588067 (+ 165us) tablet.cc:698] Row locks acquired
0110 20:58:00.588080 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:58:00.588173 (+ 93us) write_op.cc:270] Start()
0110 20:58:00.588248 (+ 75us) write_op.cc:276] Timestamp: P: 1736542680588094 usec, L: 0
0110 20:58:00.588264 (+ 16us) op_driver.cc:348] REPLICATION: starting
0110 20:58:00.588647 (+ 383us) log.cc:844] Serialized 432 byte log entry
0110 20:58:00.605838 (+ 17191us) op_driver.cc:464] REPLICATION: finished
0110 20:58:00.606211 (+ 373us) write_op.cc:301] APPLY: starting
0110 20:58:00.606259 (+ 48us) tablet.cc:1364] starting BulkCheckPresence
0110 20:58:00.606350 (+ 91us) tablet.cc:1367] finished BulkCheckPresence
0110 20:58:00.606442 (+ 92us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:58:00.606774 (+ 332us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:58:00.606901 (+ 127us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:58:00.606939 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:58:00.607093 (+ 154us) log.cc:844] Serialized 28 byte log entry
0110 20:58:00.607235 (+ 142us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:00.607295 (+ 60us) write_op.cc:454] Released schema lock
0110 20:58:00.607324 (+ 29us) write_op.cc:341] FINISH: Updating metrics
0110 20:58:00.607537 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:00.607567 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1478702,"Hive Metastore.run_cpu_time_us":555,"Hive Metastore.run_wall_time_us":188687,"child_traces":[["op",{"apply.queue_time_us":246,"apply.run_cpu_time_us":1298,"apply.run_wall_time_us":1297,"num_ops":2,"prepare.queue_time_us":768,"prepare.run_cpu_time_us":1583,"prepare.run_wall_time_us":1711,"raft.queue_time_us":73,"raft.run_cpu_time_us":10,"raft.run_wall_time_us":11,"replication_time_us":17499}]]}
I20250110 20:58:00.615926 7606 tablet_bootstrap.cc:654] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:00.623483 6992 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_34c642de1a56476a83c2b97f9cdd1287 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:58:00.630971 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 9f5d51c407bd4e67ac04c64879a18e5d (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
W20250110 20:58:00.631830 6980 catalog_manager.cc:4852] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet eff2ce38b9864782b1a1a27d05c9d777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eff2ce38b9864782b1a1a27d05c9d777 already in progress: creating tablet
I20250110 20:58:00.632529 7615 tablet_replica.cc:331] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:00.634029 7796 raft_consensus.cc:491] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:00.634510 7796 raft_consensus.cc:513] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.636219 7796 leader_election.cc:290] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:00.638362 7609 tablet_bootstrap.cc:654] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.638834 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d with delete_type TABLET_DATA_DELETED (replaced by tablet ac84d3932d5e421fa0b641419eeebe6a) from {username='slave'} at 127.0.0.1:40394
W20250110 20:58:00.639180 7287 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
W20250110 20:58:00.640508 6980 catalog_manager.cc:4839] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet 9f5d51c407bd4e67ac04c64879a18e5d because the tablet was not found. No further retry: Not found: Tablet not found: 9f5d51c407bd4e67ac04c64879a18e5d
I20250110 20:58:00.638408 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5d51c407bd4e67ac04c64879a18e5d with delete_type TABLET_DATA_DELETED (replaced by tablet ac84d3932d5e421fa0b641419eeebe6a) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:00.633241 7615 raft_consensus.cc:2238] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 20:58:00.643359 7288 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:00.643693 7288 leader_election.cc:304] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:00.643558 7615 raft_consensus.cc:2267] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.644536 7796 raft_consensus.cc:2743] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:00.646442 7615 ts_tablet_manager.cc:1905] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:58:00.647866 6980 catalog_manager.cc:4839] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet 9f5d51c407bd4e67ac04c64879a18e5d because the tablet was not found. No further retry: Not found: Tablet not found: 9f5d51c407bd4e67ac04c64879a18e5d
I20250110 20:58:00.650426 7606 tablet_bootstrap.cc:492] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:00.650797 7606 ts_tablet_manager.cc:1397] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.119s user 0.011s sys 0.001s
I20250110 20:58:00.654350 7606 raft_consensus.cc:357] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:00.654996 7606 raft_consensus.cc:383] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.655244 7606 raft_consensus.cc:738] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
W20250110 20:58:00.657318 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:00.658422 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:00.662225 7609 tablet_bootstrap.cc:492] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:00.662608 7609 ts_tablet_manager.cc:1397] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.000s
I20250110 20:58:00.664705 7609 raft_consensus.cc:357] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:00.673666 7606 consensus_queue.cc:260] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:00.683341 7606 ts_tablet_manager.cc:1428] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.029s user 0.002s sys 0.003s
I20250110 20:58:00.687412 7606 tablet_bootstrap.cc:492] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:00.665347 7609 raft_consensus.cc:383] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.691489 7609 raft_consensus.cc:738] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.691898 7602 tablet_bootstrap.cc:492] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:00.692175 7609 consensus_queue.cc:260] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:00.697479 7602 tablet_bootstrap.cc:654] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.704842 7606 tablet_bootstrap.cc:654] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.743224 7481 tablet_service.cc:1467] Processing CreateTablet for tablet e0effe5423274f66afb3fabe413427ef (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.747581 7609 ts_tablet_manager.cc:1428] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.085s user 0.005s sys 0.001s
I20250110 20:58:00.753590 7609 tablet_bootstrap.cc:492] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:00.755021 7332 tablet_service.cc:1467] Processing CreateTablet for tablet e0effe5423274f66afb3fabe413427ef (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.755520 7602 tablet_bootstrap.cc:492] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:00.755661 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0effe5423274f66afb3fabe413427ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.755908 7602 ts_tablet_manager.cc:1397] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.000s
I20250110 20:58:00.755973 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0effe5423274f66afb3fabe413427ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.758500 7602 raft_consensus.cc:357] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.760322 7602 raft_consensus.cc:383] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.760573 7602 raft_consensus.cc:738] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.761126 7602 consensus_queue.cc:260] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.764994 7796 raft_consensus.cc:491] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:00.765446 7796 raft_consensus.cc:513] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.767134 6987 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:00.767333 7796 leader_election.cc:290] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:00.768638 6987 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ac84d3932d5e421fa0b641419eeebe6a consensus state has no leader
I20250110 20:58:00.769522 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad3b006f8364950bcf60d0b0888b819" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:00.770063 7209 raft_consensus.cc:2463] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:00.771214 7287 leader_election.cc:304] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:00.772061 7796 raft_consensus.cc:2798] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:00.772426 7796 raft_consensus.cc:491] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:00.772791 7796 raft_consensus.cc:3054] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:00.772553 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad3b006f8364950bcf60d0b0888b819" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
W20250110 20:58:00.773994 7288 leader_election.cc:343] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:00.774860 6989 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_cee5a6c92d284bf2ab6f7eb541bbb600"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:00.777662 7615 ts_tablet_manager.cc:1918] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.777984 7615 log.cc:1198] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/eff2ce38b9864782b1a1a27d05c9d777
I20250110 20:58:00.778287 7609 tablet_bootstrap.cc:654] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.778539 7615 ts_tablet_manager.cc:1939] T eff2ce38b9864782b1a1a27d05c9d777 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:00.779608 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet eff2ce38b9864782b1a1a27d05c9d777 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:00.782234 7189 tablet_service.cc:1467] Processing CreateTablet for tablet e0effe5423274f66afb3fabe413427ef (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.783098 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0effe5423274f66afb3fabe413427ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.786970 6987 master_service.cc:946] ReplaceTablet: received request to replace tablet eff2ce38b9864782b1a1a27d05c9d777 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:00.789973 7796 raft_consensus.cc:513] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.793756 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 40083e396e1b4a13a77e98ade14a0d04 (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.790956 7606 tablet_bootstrap.cc:492] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:00.794476 7606 ts_tablet_manager.cc:1397] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.110s user 0.012s sys 0.004s
I20250110 20:58:00.794682 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40083e396e1b4a13a77e98ade14a0d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.795255 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 40083e396e1b4a13a77e98ade14a0d04 (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.796283 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40083e396e1b4a13a77e98ade14a0d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.796321 6987 master_service.cc:946] ReplaceTablet: received request to replace tablet ad0d5f590ed34563be385fe98a149a3f from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:00.798911 7606 raft_consensus.cc:357] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.802593 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad3b006f8364950bcf60d0b0888b819" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:00.803599 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad3b006f8364950bcf60d0b0888b819" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:00.804127 7209 raft_consensus.cc:3054] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:00.811110 7602 ts_tablet_manager.cc:1428] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20250110 20:58:00.816144 6985 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:00.818107 6986 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_4dac0491728e4031866a732faeb6d42c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:00.822589 7209 raft_consensus.cc:2463] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
W20250110 20:58:00.805933 7288 leader_election.cc:343] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:00.825922 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 40083e396e1b4a13a77e98ade14a0d04 (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.826779 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40083e396e1b4a13a77e98ade14a0d04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.828173 7287 leader_election.cc:304] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:00.828876 7863 raft_consensus.cc:2798] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:00.829280 7796 leader_election.cc:290] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:58:00.828939 7606 raft_consensus.cc:383] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.833906 7606 raft_consensus.cc:738] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.838191 7606 consensus_queue.cc:260] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.846920 7863 raft_consensus.cc:695] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:00.848400 7863 consensus_queue.cc:237] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.847719 6987 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet ad0d5f590ed34563be385fe98a149a3f
I20250110 20:58:00.854456 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0d5f590ed34563be385fe98a149a3f with delete_type TABLET_DATA_DELETED (replaced by tablet 77661375951e456899d2f9052dc680e7) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:00.855444 7616 tablet_replica.cc:331] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:00.856083 7616 raft_consensus.cc:2238] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.856566 7616 raft_consensus.cc:2267] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.858986 7616 ts_tablet_manager.cc:1905] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.861382 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0d5f590ed34563be385fe98a149a3f with delete_type TABLET_DATA_DELETED (replaced by tablet 77661375951e456899d2f9052dc680e7) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:00.862728 7606 ts_tablet_manager.cc:1428] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.068s user 0.001s sys 0.004s
I20250110 20:58:00.864454 6987 catalog_manager.cc:6596] ReplaceTablet: tablet ad0d5f590ed34563be385fe98a149a3f deleted and replaced by tablet 77661375951e456899d2f9052dc680e7
I20250110 20:58:00.864162 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet ad0d5f590ed34563be385fe98a149a3f with delete_type TABLET_DATA_DELETED (replaced by tablet 77661375951e456899d2f9052dc680e7) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:00.867187 7617 tablet_replica.cc:331] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:00.867836 7617 raft_consensus.cc:2238] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.868237 7617 raft_consensus.cc:2267] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.869088 7188 tablet_service.cc:1467] Processing CreateTablet for tablet f25437547b454eafbcf56e2b6c3b348a (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
W20250110 20:58:00.869781 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ac84d3932d5e421fa0b641419eeebe6a consensus state has no leader
I20250110 20:58:00.870141 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f25437547b454eafbcf56e2b6c3b348a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.870157 7606 tablet_bootstrap.cc:492] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:00.870446 7617 ts_tablet_manager.cc:1905] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.876470 7868 raft_consensus.cc:491] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:00.876881 7868 raft_consensus.cc:513] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.878700 7868 leader_election.cc:290] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:00.880100 7332 tablet_service.cc:1467] Processing CreateTablet for tablet f25437547b454eafbcf56e2b6c3b348a (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.880991 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f25437547b454eafbcf56e2b6c3b348a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.885308 7602 tablet_bootstrap.cc:492] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:00.886344 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac84d3932d5e421fa0b641419eeebe6a" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:00.886607 7481 tablet_service.cc:1467] Processing CreateTablet for tablet f25437547b454eafbcf56e2b6c3b348a (DEFAULT_TABLE table=default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.887574 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f25437547b454eafbcf56e2b6c3b348a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:00.887521 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac84d3932d5e421fa0b641419eeebe6a" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
W20250110 20:58:00.888696 7145 leader_election.cc:343] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:58:00.889485 7145 leader_election.cc:343] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:00.889817 7145 leader_election.cc:304] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:00.891577 7868 raft_consensus.cc:2743] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:00.892870 7617 ts_tablet_manager.cc:1918] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.897218 7617 log.cc:1198] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ad0d5f590ed34563be385fe98a149a3f
I20250110 20:58:00.898173 7617 ts_tablet_manager.cc:1939] T ad0d5f590ed34563be385fe98a149a3f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:00.903666 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet ad0d5f590ed34563be385fe98a149a3f (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:00.903982 7609 tablet_bootstrap.cc:492] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:00.904357 7609 ts_tablet_manager.cc:1397] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.155s user 0.019s sys 0.018s
I20250110 20:58:00.906812 7615 tablet_replica.cc:331] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
W20250110 20:58:00.906886 6979 catalog_manager.cc:4839] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet eff2ce38b9864782b1a1a27d05c9d777 because the tablet was not found. No further retry: Not found: Tablet not found: eff2ce38b9864782b1a1a27d05c9d777
I20250110 20:58:00.907495 7615 raft_consensus.cc:2238] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:00.907898 7615 raft_consensus.cc:2267] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:00.914929 7606 tablet_bootstrap.cc:654] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.909386 7615 ts_tablet_manager.cc:1905] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:00.923332 7609 raft_consensus.cc:357] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.924151 7609 raft_consensus.cc:383] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.924412 7609 raft_consensus.cc:738] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.910720 7602 tablet_bootstrap.cc:654] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:00.924991 7609 consensus_queue.cc:260] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:00.947058 7616 ts_tablet_manager.cc:1918] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:00.947479 7616 log.cc:1198] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ad0d5f590ed34563be385fe98a149a3f
I20250110 20:58:00.948210 7616 ts_tablet_manager.cc:1939] T ad0d5f590ed34563be385fe98a149a3f P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:00.950600 7606 tablet_bootstrap.cc:492] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:00.950999 7606 ts_tablet_manager.cc:1397] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.001s
I20250110 20:58:00.951557 6987 catalog_manager.cc:5526] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:00.953294 7606 raft_consensus.cc:357] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.953933 7606 raft_consensus.cc:383] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:00.954206 7606 raft_consensus.cc:738] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:00.954799 7606 consensus_queue.cc:260] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:00.959959 7606 ts_tablet_manager.cc:1428] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20250110 20:58:00.966835 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet ad0d5f590ed34563be385fe98a149a3f (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:00.969396 7609 ts_tablet_manager.cc:1428] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.065s user 0.008s sys 0.004s
I20250110 20:58:00.978832 7609 tablet_bootstrap.cc:492] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:00.985862 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 77661375951e456899d2f9052dc680e7 (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:00.989200 7606 tablet_bootstrap.cc:492] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:01.008250 6987 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:01.008203 7606 tablet_bootstrap.cc:654] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.013772 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77661375951e456899d2f9052dc680e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.019651 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 with delete_type TABLET_DATA_DELETED (replaced by tablet f25437547b454eafbcf56e2b6c3b348a) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:01.020411 7617 tablet_replica.cc:331] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:01.021133 7617 raft_consensus.cc:2238] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:01.021584 7617 raft_consensus.cc:2267] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.022068 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet ad0d5f590ed34563be385fe98a149a3f from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:01.023665 7617 ts_tablet_manager.cc:1905] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.026995 6985 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:01.033314 7602 tablet_bootstrap.cc:492] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:01.035072 7602 ts_tablet_manager.cc:1397] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.000s
I20250110 20:58:01.035753 7352 raft_consensus.cc:3054] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.040563 7609 tablet_bootstrap.cc:654] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.050184 7606 tablet_bootstrap.cc:492] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:01.050540 7606 ts_tablet_manager.cc:1397] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.004s
I20250110 20:58:01.051689 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 77661375951e456899d2f9052dc680e7 (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.052598 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77661375951e456899d2f9052dc680e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.038609 7602 raft_consensus.cc:357] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.059475 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 77661375951e456899d2f9052dc680e7 (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.060472 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77661375951e456899d2f9052dc680e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.052582 7606 raft_consensus.cc:357] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.065507 7606 raft_consensus.cc:383] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.065742 7606 raft_consensus.cc:738] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.066304 7606 consensus_queue.cc:260] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.064476 7907 tablet.cc:1721] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:01.079392 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:01.080770 7609 tablet_bootstrap.cc:492] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:01.083259 7609 ts_tablet_manager.cc:1397] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.001s
I20250110 20:58:01.083518 6987 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]). Expected version 1 got 0
I20250110 20:58:01.084933 6988 catalog_manager.cc:3422] renamed table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376] in HMS: new name default.table_176fb71364f24095a1efbf0d29c18ff8
I20250110 20:58:01.086539 7609 raft_consensus.cc:357] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.078876 7606 ts_tablet_manager.cc:1428] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.028s user 0.005s sys 0.000s
I20250110 20:58:01.087880 7606 tablet_bootstrap.cc:492] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:01.094290 7602 raft_consensus.cc:383] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.094580 7602 raft_consensus.cc:738] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.094934 7609 raft_consensus.cc:383] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.095244 7617 ts_tablet_manager.cc:1918] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:01.095602 7617 log.cc:1198] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/eff2ce38b9864782b1a1a27d05c9d777
I20250110 20:58:01.096266 7617 ts_tablet_manager.cc:1939] T eff2ce38b9864782b1a1a27d05c9d777 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:01.095258 7602 consensus_queue.cc:260] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.095598 7609 raft_consensus.cc:738] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.096989 7609 consensus_queue.cc:260] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.103721 7602 ts_tablet_manager.cc:1428] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.067s user 0.004s sys 0.000s
I20250110 20:58:01.110793 7609 ts_tablet_manager.cc:1428] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20250110 20:58:01.113709 7606 tablet_bootstrap.cc:654] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.116045 7609 tablet_bootstrap.cc:492] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:01.134788 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet eff2ce38b9864782b1a1a27d05c9d777 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:01.138907 7749 consensus_queue.cc:1035] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:58:01.152911 7609 tablet_bootstrap.cc:654] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.166772 7749 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:01.167197 7602 tablet_bootstrap.cc:492] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:01.167202 7749 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.169057 7749 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:01.171609 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:01.179309 7609 tablet_bootstrap.cc:492] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:01.197695 7602 tablet_bootstrap.cc:654] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.205452 7615 ts_tablet_manager.cc:1918] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250110 20:58:01.189910 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:01.207664 7436 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:01.210858 7749 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:01.211408 7749 consensus_queue.cc:1035] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:01.190342 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet eff2ce38b9864782b1a1a27d05c9d777 with delete_type TABLET_DATA_DELETED (replaced by tablet f25437547b454eafbcf56e2b6c3b348a) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:01.207819 7609 ts_tablet_manager.cc:1397] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.001s
I20250110 20:58:01.212787 7616 tablet_replica.cc:331] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:01.213590 7616 raft_consensus.cc:2238] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:01.214005 7616 raft_consensus.cc:2267] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.214634 7606 tablet_bootstrap.cc:492] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:01.214990 7606 ts_tablet_manager.cc:1397] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.003s
I20250110 20:58:01.216208 7616 ts_tablet_manager.cc:1905] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.218061 7606 raft_consensus.cc:357] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.218703 7606 raft_consensus.cc:383] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.218947 7606 raft_consensus.cc:738] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.205801 7615 log.cc:1198] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ad0d5f590ed34563be385fe98a149a3f
I20250110 20:58:01.236416 7615 ts_tablet_manager.cc:1939] T ad0d5f590ed34563be385fe98a149a3f P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:01.240271 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet ad0d5f590ed34563be385fe98a149a3f (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:01.242542 7609 raft_consensus.cc:357] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.243389 7609 raft_consensus.cc:383] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.243649 7609 raft_consensus.cc:738] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.244310 7609 consensus_queue.cc:260] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:58:01.259824 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet eff2ce38b9864782b1a1a27d05c9d777
I20250110 20:58:01.259969 7616 ts_tablet_manager.cc:1918] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:01.260428 7616 log.cc:1198] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/eff2ce38b9864782b1a1a27d05c9d777
I20250110 20:58:01.261111 7616 ts_tablet_manager.cc:1939] T eff2ce38b9864782b1a1a27d05c9d777 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:01.263345 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet eff2ce38b9864782b1a1a27d05c9d777 (table default.table_045d0574736e4b9e8fdf9e6a91066d06 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:01.264295 7609 ts_tablet_manager.cc:1428] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.051s user 0.006s sys 0.000s
I20250110 20:58:01.269275 7609 tablet_bootstrap.cc:492] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:01.273679 7606 consensus_queue.cc:260] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.278062 7863 raft_consensus.cc:491] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:01.278492 7863 raft_consensus.cc:513] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.280138 7863 leader_election.cc:290] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:58:01.281585 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40083e396e1b4a13a77e98ade14a0d04" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:01.282176 7210 raft_consensus.cc:2463] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:01.282667 7609 tablet_bootstrap.cc:654] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.283278 7287 leader_election.cc:304] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:01.285368 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40083e396e1b4a13a77e98ade14a0d04" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
W20250110 20:58:01.287487 7288 leader_election.cc:343] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:01.293748 7606 ts_tablet_manager.cc:1428] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.078s user 0.005s sys 0.000s
I20250110 20:58:01.293975 7863 raft_consensus.cc:2798] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:01.294298 7863 raft_consensus.cc:491] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:01.294593 7863 raft_consensus.cc:3054] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.301733 7863 raft_consensus.cc:513] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.303184 6987 master_service.cc:946] ReplaceTablet: received request to replace tablet e0effe5423274f66afb3fabe413427ef from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:01.303709 7863 leader_election.cc:290] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:58:01.304644 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40083e396e1b4a13a77e98ade14a0d04" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:01.305145 7210 raft_consensus.cc:3054] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.306885 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40083e396e1b4a13a77e98ade14a0d04" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
W20250110 20:58:01.312585 7288 leader_election.cc:343] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:01.313388 7210 raft_consensus.cc:2463] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:01.314424 7287 leader_election.cc:304] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:01.316061 7606 tablet_bootstrap.cc:492] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:01.322719 7602 tablet_bootstrap.cc:492] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:01.323118 7602 ts_tablet_manager.cc:1397] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.156s user 0.007s sys 0.001s
I20250110 20:58:01.325168 7602 raft_consensus.cc:357] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.325587 7863 raft_consensus.cc:2798] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:01.325795 7602 raft_consensus.cc:383] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.326040 7602 raft_consensus.cc:738] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.326036 7863 raft_consensus.cc:695] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:01.327461 7602 consensus_queue.cc:260] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.327569 7863 consensus_queue.cc:237] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.336189 7609 tablet_bootstrap.cc:492] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:01.336614 7609 ts_tablet_manager.cc:1397] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.068s user 0.008s sys 0.000s
I20250110 20:58:01.336861 7868 raft_consensus.cc:491] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:01.337261 7868 raft_consensus.cc:513] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.338951 7868 leader_election.cc:290] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:01.338956 7609 raft_consensus.cc:357] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.340863 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0effe5423274f66afb3fabe413427ef" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:01.341753 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0effe5423274f66afb3fabe413427ef" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:01.342160 7602 ts_tablet_manager.cc:1428] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250110 20:58:01.342365 7352 raft_consensus.cc:2463] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:01.343768 7145 leader_election.cc:304] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:01.345230 7868 raft_consensus.cc:2798] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:01.345553 7868 raft_consensus.cc:491] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:01.345847 7868 raft_consensus.cc:3054] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.354897 7868 raft_consensus.cc:513] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.339592 7609 raft_consensus.cc:383] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.355532 7609 raft_consensus.cc:738] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.357108 7868 leader_election.cc:290] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:01.357439 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0effe5423274f66afb3fabe413427ef" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:01.359061 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0effe5423274f66afb3fabe413427ef" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:01.359583 7352 raft_consensus.cc:3054] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.366176 7352 raft_consensus.cc:2463] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:01.356169 7609 consensus_queue.cc:260] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.367655 7501 raft_consensus.cc:2463] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:01.375059 7145 leader_election.cc:304] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:01.377063 7609 ts_tablet_manager.cc:1428] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.039s user 0.007s sys 0.000s
I20250110 20:58:01.388595 7609 tablet_bootstrap.cc:492] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:01.392746 6987 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet e0effe5423274f66afb3fabe413427ef
I20250110 20:58:01.394251 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet e0effe5423274f66afb3fabe413427ef with delete_type TABLET_DATA_DELETED (replaced by tablet ec62f901ffe64faea7d3716db20c487f) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:01.394850 7616 tablet_replica.cc:331] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:01.395488 7616 raft_consensus.cc:2238] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:01.395864 7616 raft_consensus.cc:2267] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.400699 7616 ts_tablet_manager.cc:1905] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.400599 6985 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:01.402139 7868 raft_consensus.cc:2798] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:01.403465 7606 tablet_bootstrap.cc:654] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.406328 7602 tablet_bootstrap.cc:492] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:01.411938 7602 tablet_bootstrap.cc:654] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.405983 7609 tablet_bootstrap.cc:654] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.417495 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet e0effe5423274f66afb3fabe413427ef with delete_type TABLET_DATA_DELETED (replaced by tablet ec62f901ffe64faea7d3716db20c487f) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:01.418668 7617 tablet_replica.cc:331] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:01.418670 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet e0effe5423274f66afb3fabe413427ef with delete_type TABLET_DATA_DELETED (replaced by tablet ec62f901ffe64faea7d3716db20c487f) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:01.419307 7615 tablet_replica.cc:331] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:01.419299 7616 ts_tablet_manager.cc:1918] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:01.419365 7617 raft_consensus.cc:2238] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:01.419754 7617 raft_consensus.cc:2267] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.420884 6987 catalog_manager.cc:6596] ReplaceTablet: tablet e0effe5423274f66afb3fabe413427ef deleted and replaced by tablet ec62f901ffe64faea7d3716db20c487f
I20250110 20:58:01.421645 7617 ts_tablet_manager.cc:1905] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.420257 7616 log.cc:1198] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/e0effe5423274f66afb3fabe413427ef
I20250110 20:58:01.423003 7616 ts_tablet_manager.cc:1939] T e0effe5423274f66afb3fabe413427ef P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:01.423241 6990 catalog_manager.cc:3422] renamed table default.table_391e103e657c4fa2914f89bf58d900be [id=ca7388d37c1f4aa8ac6ff138c21c7047] in HMS: new name default.table_ba5d6def0428488589bda04057a79fab
I20250110 20:58:01.425755 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet e0effe5423274f66afb3fabe413427ef (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]) successfully deleted
I20250110 20:58:01.431841 7868 raft_consensus.cc:695] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:58:01.432750 7868 consensus_queue.cc:237] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.451133 7606 tablet_bootstrap.cc:492] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:01.451515 7606 ts_tablet_manager.cc:1397] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.136s user 0.008s sys 0.000s
I20250110 20:58:01.454432 7606 raft_consensus.cc:357] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.455054 7606 raft_consensus.cc:383] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.455312 7606 raft_consensus.cc:738] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.465101 7617 ts_tablet_manager.cc:1918] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:01.465476 7617 log.cc:1198] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/e0effe5423274f66afb3fabe413427ef
I20250110 20:58:01.466064 7617 ts_tablet_manager.cc:1939] T e0effe5423274f66afb3fabe413427ef P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:01.466172 7609 tablet_bootstrap.cc:492] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:01.466678 7609 ts_tablet_manager.cc:1397] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.078s user 0.014s sys 0.001s
I20250110 20:58:01.468210 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet e0effe5423274f66afb3fabe413427ef (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]) successfully deleted
I20250110 20:58:01.468992 7609 raft_consensus.cc:357] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.469720 7609 raft_consensus.cc:383] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.470017 7609 raft_consensus.cc:738] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.476348 7615 raft_consensus.cc:2238] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:01.476881 7609 consensus_queue.cc:260] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.477478 7615 pending_rounds.cc:62] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:58:01.477698 7615 pending_rounds.cc:69] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878823158165504 op_type: NO_OP noop_request { }
I20250110 20:58:01.478016 7615 raft_consensus.cc:2883] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:01.478312 7615 raft_consensus.cc:2267] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.486042 7609 ts_tablet_manager.cc:1428] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20250110 20:58:01.491940 7615 ts_tablet_manager.cc:1905] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.495393 7914 raft_consensus.cc:491] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:01.495824 7914 raft_consensus.cc:513] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:01.495958 7606 consensus_queue.cc:260] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.497675 7914 leader_election.cc:290] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:58:01.499419 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad3b006f8364950bcf60d0b0888b819" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:01.500134 7210 raft_consensus.cc:2388] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c3b63d95ca734d8a9803b96bfd94f307 in current term 1: Already voted for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in this term.
I20250110 20:58:01.502924 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad3b006f8364950bcf60d0b0888b819" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:01.505336 7436 leader_election.cc:304] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:01.506551 7914 raft_consensus.cc:3054] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
W20250110 20:58:01.510564 7049 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f25437547b454eafbcf56e2b6c3b348a consensus state has no leader
I20250110 20:58:01.513979 7914 raft_consensus.cc:2743] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:01.514807 7609 tablet_bootstrap.cc:492] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:01.526070 7615 ts_tablet_manager.cc:1918] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:01.526422 7615 log.cc:1198] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/e0effe5423274f66afb3fabe413427ef
I20250110 20:58:01.527591 6992 catalog_manager.cc:2149] created HMS catalog entry for table [id=f089167c681940c1a2f841509f560366]
I20250110 20:58:01.531250 7615 ts_tablet_manager.cc:1939] T e0effe5423274f66afb3fabe413427ef P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:01.534247 7188 tablet_service.cc:1467] Processing CreateTablet for tablet ec62f901ffe64faea7d3716db20c487f (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.535266 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec62f901ffe64faea7d3716db20c487f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.537091 7609 tablet_bootstrap.cc:654] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.537654 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet e0effe5423274f66afb3fabe413427ef (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]) successfully deleted
I20250110 20:58:01.541952 7863 raft_consensus.cc:491] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:01.542393 7863 raft_consensus.cc:513] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.544195 7863 leader_election.cc:290] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:01.545837 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac84d3932d5e421fa0b641419eeebe6a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:01.546413 7210 raft_consensus.cc:2463] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:01.547291 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac84d3932d5e421fa0b641419eeebe6a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:01.547863 7501 raft_consensus.cc:2463] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:01.549054 7288 leader_election.cc:304] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:01.550163 7863 raft_consensus.cc:2798] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:01.550516 7863 raft_consensus.cc:491] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:01.550895 7863 raft_consensus.cc:3054] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.552870 7481 tablet_service.cc:1467] Processing CreateTablet for tablet ec62f901ffe64faea7d3716db20c487f (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.553568 7332 tablet_service.cc:1467] Processing CreateTablet for tablet ec62f901ffe64faea7d3716db20c487f (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.553818 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec62f901ffe64faea7d3716db20c487f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.554443 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec62f901ffe64faea7d3716db20c487f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.563638 7863 raft_consensus.cc:513] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.566388 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac84d3932d5e421fa0b641419eeebe6a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:01.566936 7210 raft_consensus.cc:3054] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.566390 6985 catalog_manager.cc:5526] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:01.570189 7602 tablet_bootstrap.cc:492] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:01.573642 7210 raft_consensus.cc:2463] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:01.574703 7287 leader_election.cc:304] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:01.575466 7897 raft_consensus.cc:2798] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:01.583022 7609 tablet_bootstrap.cc:492] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:01.583348 7606 ts_tablet_manager.cc:1428] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.132s user 0.006s sys 0.000s
I20250110 20:58:01.570561 7602 ts_tablet_manager.cc:1397] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.164s user 0.004s sys 0.004s
I20250110 20:58:01.585964 7602 raft_consensus.cc:357] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.586586 7602 raft_consensus.cc:383] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.586827 7602 raft_consensus.cc:738] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.587373 7602 consensus_queue.cc:260] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.602066 7863 leader_election.cc:290] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:01.588789 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac84d3932d5e421fa0b641419eeebe6a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:01.603772 7501 raft_consensus.cc:3054] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.606868 7606 tablet_bootstrap.cc:492] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:01.610443 7501 raft_consensus.cc:2463] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:01.611382 7609 ts_tablet_manager.cc:1397] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.097s user 0.009s sys 0.002s
I20250110 20:58:01.620115 7897 raft_consensus.cc:695] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:01.621126 7897 consensus_queue.cc:237] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.637643 6992 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 70) took 2076 ms. Trace:
I20250110 20:58:01.637991 6992 rpcz_store.cc:276] 0110 20:57:59.561168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:59.568719 (+ 7551us) service_pool.cc:225] Handling call
0110 20:58:00.623876 (+1055157us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:58:00.627155 (+ 3279us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:58:01.527545 (+900390us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:01.636944 (+109399us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:01.637274 (+ 330us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:01.637586 (+ 312us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:58:01.529634 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:58:01.530025 (+ 391us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:58:01.530048 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:58:01.530062 (+ 14us) tablet.cc:659] Decoding operations
0110 20:58:01.530324 (+ 262us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:58:01.530365 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0110 20:58:01.530387 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:58:01.530577 (+ 190us) tablet.cc:698] Row locks acquired
0110 20:58:01.530591 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:58:01.530696 (+ 105us) write_op.cc:270] Start()
0110 20:58:01.530778 (+ 82us) write_op.cc:276] Timestamp: P: 1736542681530622 usec, L: 0
0110 20:58:01.530794 (+ 16us) op_driver.cc:348] REPLICATION: starting
0110 20:58:01.607200 (+ 76406us) log.cc:844] Serialized 433 byte log entry
0110 20:58:01.607789 (+ 589us) spinlock_profiling.cc:243] Waited 75.9 ms on lock 0x7b54000316d4. stack: 00007f728d56d853 00007f728d56d750 00007f728d0e2584 00000000004fcd24 00000000004fccb9 00007f729446d6d1 00007f72944e4198 00007f72949503b1 00007f729494fb88 00007f7294954821 00007f72949547d9 00007f7294954769 00007f7294954731 00007f729495395d 000000000059ea04 000000000059e839
0110 20:58:01.622201 (+ 14412us) op_driver.cc:464] REPLICATION: finished
0110 20:58:01.635490 (+ 13289us) write_op.cc:301] APPLY: starting
0110 20:58:01.635545 (+ 55us) tablet.cc:1364] starting BulkCheckPresence
0110 20:58:01.635643 (+ 98us) tablet.cc:1367] finished BulkCheckPresence
0110 20:58:01.635661 (+ 18us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:58:01.636007 (+ 346us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:58:01.636143 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:58:01.636186 (+ 43us) write_op.cc:312] APPLY: finished
0110 20:58:01.636354 (+ 168us) log.cc:844] Serialized 29 byte log entry
0110 20:58:01.636506 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:01.636565 (+ 59us) write_op.cc:454] Released schema lock
0110 20:58:01.636597 (+ 32us) write_op.cc:341] FINISH: Updating metrics
0110 20:58:01.636795 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:01.636817 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":802609,"Hive Metastore.run_cpu_time_us":550,"Hive Metastore.run_wall_time_us":96880,"child_traces":[["op",{"apply.queue_time_us":13130,"apply.run_cpu_time_us":1280,"apply.run_wall_time_us":1280,"num_ops":2,"prepare.queue_time_us":328,"prepare.run_cpu_time_us":3259,"prepare.run_wall_time_us":78390,"raft.queue_time_us":313,"raft.run_cpu_time_us":520,"raft.run_wall_time_us":520,"replication_time_us":91330,"spinlock_wait_cycles":174628608}]]}
I20250110 20:58:01.633462 7609 raft_consensus.cc:357] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.643469 7609 raft_consensus.cc:383] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.644659 7609 raft_consensus.cc:738] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.645452 7609 consensus_queue.cc:260] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.654613 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet e0effe5423274f66afb3fabe413427ef with delete_type TABLET_DATA_DELETED (replaced by tablet ec62f901ffe64faea7d3716db20c487f) from {username='slave'} at 127.0.0.1:42840
W20250110 20:58:01.656327 6979 catalog_manager.cc:4839] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet e0effe5423274f66afb3fabe413427ef because the tablet was not found. No further retry: Not found: Tablet not found: e0effe5423274f66afb3fabe413427ef
I20250110 20:58:01.657078 6992 master_service.cc:946] ReplaceTablet: received request to replace tablet 6ad3b006f8364950bcf60d0b0888b819 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:01.659778 7609 ts_tablet_manager.cc:1428] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.042s user 0.005s sys 0.000s
I20250110 20:58:01.674726 7609 tablet_bootstrap.cc:492] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:01.677170 7932 tablet.cc:1721] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250110 20:58:01.685837 6993 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_e760ae0763c04467a8fea38edff37362 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:58:01.691766 6987 master_service.cc:946] ReplaceTablet: received request to replace tablet 40083e396e1b4a13a77e98ade14a0d04 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:01.697091 6992 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 6ad3b006f8364950bcf60d0b0888b819
I20250110 20:58:01.698887 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad3b006f8364950bcf60d0b0888b819 with delete_type TABLET_DATA_DELETED (replaced by tablet f3e994b6ed3e4692a4ba3cfcf3b72648) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:01.699570 7617 tablet_replica.cc:331] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:01.699914 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad3b006f8364950bcf60d0b0888b819 with delete_type TABLET_DATA_DELETED (replaced by tablet f3e994b6ed3e4692a4ba3cfcf3b72648) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:01.700246 7617 raft_consensus.cc:2238] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:01.700568 7615 tablet_replica.cc:331] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:01.700838 7617 pending_rounds.cc:62] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:58:01.701056 7617 pending_rounds.cc:69] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878820764401664 op_type: NO_OP noop_request { }
I20250110 20:58:01.701256 7615 raft_consensus.cc:2238] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:01.701395 7617 raft_consensus.cc:2883] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:01.701675 7615 raft_consensus.cc:2267] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.701709 7617 raft_consensus.cc:2267] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.703365 7602 ts_tablet_manager.cc:1428] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.119s user 0.004s sys 0.000s
I20250110 20:58:01.703601 7615 ts_tablet_manager.cc:1905] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.703652 7617 ts_tablet_manager.cc:1905] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.707684 6992 catalog_manager.cc:6596] ReplaceTablet: tablet 6ad3b006f8364950bcf60d0b0888b819 deleted and replaced by tablet f3e994b6ed3e4692a4ba3cfcf3b72648
I20250110 20:58:01.716307 6991 catalog_manager.cc:2686] deleted HMS catalog entry for table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]
I20250110 20:58:01.698282 7933 tablet.cc:1721] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:01.718189 6992 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:01.718797 7897 raft_consensus.cc:491] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:01.719214 7897 raft_consensus.cc:513] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.721344 7897 leader_election.cc:290] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:01.722110 7606 tablet_bootstrap.cc:654] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.722483 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25437547b454eafbcf56e2b6c3b348a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:01.723085 7501 raft_consensus.cc:2463] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:01.724416 7288 leader_election.cc:304] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:01.725208 7897 raft_consensus.cc:2798] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:01.725589 7897 raft_consensus.cc:491] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:01.725956 7897 raft_consensus.cc:3054] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.729758 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25437547b454eafbcf56e2b6c3b348a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:01.730432 7210 raft_consensus.cc:2463] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:01.731913 7602 tablet_bootstrap.cc:492] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:01.735134 7897 raft_consensus.cc:513] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.731040 6987 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 40083e396e1b4a13a77e98ade14a0d04
I20250110 20:58:01.739845 6987 catalog_manager.cc:6596] ReplaceTablet: tablet 40083e396e1b4a13a77e98ade14a0d04 deleted and replaced by tablet 9f62814cd3544ba6a1bbe0550dd8734c
I20250110 20:58:01.740926 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 40083e396e1b4a13a77e98ade14a0d04 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f62814cd3544ba6a1bbe0550dd8734c) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:01.742501 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 40083e396e1b4a13a77e98ade14a0d04 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f62814cd3544ba6a1bbe0550dd8734c) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:01.743155 7616 tablet_replica.cc:331] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
W20250110 20:58:01.743727 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:01.743805 7616 raft_consensus.cc:2238] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250110 20:58:01.744164 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:01.744237 7616 raft_consensus.cc:2267] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.745565 6992 catalog_manager.cc:5526] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:01.746847 6992 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:01.749770 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet 40083e396e1b4a13a77e98ade14a0d04 with delete_type TABLET_DATA_DELETED (replaced by tablet 9f62814cd3544ba6a1bbe0550dd8734c) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:01.750326 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad3b006f8364950bcf60d0b0888b819 with delete_type TABLET_DATA_DELETED (replaced by tablet f3e994b6ed3e4692a4ba3cfcf3b72648) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:01.759148 7616 ts_tablet_manager.cc:1905] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.767360 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25437547b454eafbcf56e2b6c3b348a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:01.768008 7210 raft_consensus.cc:3054] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.768193 7609 tablet_bootstrap.cc:654] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.769692 7931 tablet.cc:1721] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:01.772497 7606 tablet_bootstrap.cc:492] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:01.772897 7606 ts_tablet_manager.cc:1397] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.166s user 0.005s sys 0.003s
I20250110 20:58:01.772786 6985 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:01.774116 6987 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:01.774715 6987 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 53bbd84258e04c76a2d1bf0af353444b (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:01.774930 7210 raft_consensus.cc:2463] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:01.775080 7606 raft_consensus.cc:357] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.775712 7606 raft_consensus.cc:383] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.775946 7606 raft_consensus.cc:738] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.776105 7287 leader_election.cc:304] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:01.776991 7926 raft_consensus.cc:2798] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:01.777467 7926 raft_consensus.cc:695] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:01.778180 7926 consensus_queue.cc:237] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.783650 7602 tablet_bootstrap.cc:654] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.787417 6985 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:01.789202 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f25437547b454eafbcf56e2b6c3b348a" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:01.789762 7501 raft_consensus.cc:3054] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:01.795385 7501 raft_consensus.cc:2463] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:01.797487 7615 ts_tablet_manager.cc:1918] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:01.797850 7615 log.cc:1198] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/6ad3b006f8364950bcf60d0b0888b819
I20250110 20:58:01.798429 7615 ts_tablet_manager.cc:1939] T 6ad3b006f8364950bcf60d0b0888b819 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:01.800496 7606 consensus_queue.cc:260] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.800971 7616 ts_tablet_manager.cc:1918] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:01.810685 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 6ad3b006f8364950bcf60d0b0888b819 (table default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]) successfully deleted
I20250110 20:58:01.813684 7615 tablet_replica.cc:331] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:01.816673 7897 leader_election.cc:290] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:01.818384 7602 tablet_bootstrap.cc:492] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:01.818763 7602 ts_tablet_manager.cc:1397] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.001s
I20250110 20:58:01.820924 7602 raft_consensus.cc:357] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.821563 7602 raft_consensus.cc:383] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.821794 7602 raft_consensus.cc:738] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.824307 7617 ts_tablet_manager.cc:1918] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:01.801318 7616 log.cc:1198] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/40083e396e1b4a13a77e98ade14a0d04
I20250110 20:58:01.827205 7616 ts_tablet_manager.cc:1939] T 40083e396e1b4a13a77e98ade14a0d04 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:01.822340 7602 consensus_queue.cc:260] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.829525 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 40083e396e1b4a13a77e98ade14a0d04 (table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]) successfully deleted
I20250110 20:58:01.829989 7617 log.cc:1198] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/6ad3b006f8364950bcf60d0b0888b819
I20250110 20:58:01.834436 6992 catalog_manager.cc:6071] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047] alter complete (version 1)
I20250110 20:58:01.835543 7602 ts_tablet_manager.cc:1428] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.017s user 0.003s sys 0.003s
I20250110 20:58:01.836350 7602 tablet_bootstrap.cc:492] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:01.842288 7609 tablet_bootstrap.cc:492] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:01.827433 7606 ts_tablet_manager.cc:1428] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.054s user 0.006s sys 0.001s
I20250110 20:58:01.842705 7609 ts_tablet_manager.cc:1397] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.169s user 0.012s sys 0.005s
I20250110 20:58:01.814359 7615 raft_consensus.cc:2238] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:01.847913 7615 raft_consensus.cc:2267] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.842463 7617 ts_tablet_manager.cc:1939] T 6ad3b006f8364950bcf60d0b0888b819 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:01.849524 7615 ts_tablet_manager.cc:1905] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.846032 7606 tablet_bootstrap.cc:492] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:01.849756 7616 tablet_replica.cc:331] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:01.849669 6987 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_eed9317ae3914199a2cdc0d7e112371f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:01.850438 7616 raft_consensus.cc:2238] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:01.850870 7616 raft_consensus.cc:2267] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.852499 7616 ts_tablet_manager.cc:1905] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.854516 7609 raft_consensus.cc:357] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.855191 7609 raft_consensus.cc:383] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.855433 7609 raft_consensus.cc:738] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.856014 7609 consensus_queue.cc:260] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.856602 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 6ad3b006f8364950bcf60d0b0888b819 (table default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]) successfully deleted
I20250110 20:58:01.859731 7617 tablet_replica.cc:331] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:01.861932 7617 raft_consensus.cc:2238] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:01.872466 7609 ts_tablet_manager.cc:1428] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.030s user 0.004s sys 0.002s
I20250110 20:58:01.876466 7602 tablet_bootstrap.cc:654] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:01.884215 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f25437547b454eafbcf56e2b6c3b348a consensus state has no leader
I20250110 20:58:01.885592 7609 tablet_bootstrap.cc:492] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:01.889369 7615 ts_tablet_manager.cc:1918] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:01.892175 7615 log.cc:1198] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/40083e396e1b4a13a77e98ade14a0d04
I20250110 20:58:01.893178 7615 ts_tablet_manager.cc:1939] T 40083e396e1b4a13a77e98ade14a0d04 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:01.896306 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 40083e396e1b4a13a77e98ade14a0d04 (table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]) successfully deleted
I20250110 20:58:01.900417 6985 catalog_manager.cc:5526] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:01.901654 6985 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:01.901883 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 721c67636cbc4fe98742a00b970f3701 (DEFAULT_TABLE table=default.table_34c642de1a56476a83c2b97f9cdd1287 [id=f089167c681940c1a2f841509f560366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.902880 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 721c67636cbc4fe98742a00b970f3701. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.905539 7617 pending_rounds.cc:62] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Trying to abort 1 pending ops.
I20250110 20:58:01.905768 7617 pending_rounds.cc:69] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878822786633728 op_type: NO_OP noop_request { }
I20250110 20:58:01.906100 7617 raft_consensus.cc:2883] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:01.906419 7617 raft_consensus.cc:2267] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:01.906276 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 721c67636cbc4fe98742a00b970f3701 (DEFAULT_TABLE table=default.table_34c642de1a56476a83c2b97f9cdd1287 [id=f089167c681940c1a2f841509f560366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.862310 7606 tablet_bootstrap.cc:654] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.902168 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 721c67636cbc4fe98742a00b970f3701 (DEFAULT_TABLE table=default.table_34c642de1a56476a83c2b97f9cdd1287 [id=f089167c681940c1a2f841509f560366]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.907181 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 721c67636cbc4fe98742a00b970f3701. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.907613 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 721c67636cbc4fe98742a00b970f3701. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.920841 6992 master_service.cc:946] ReplaceTablet: received request to replace tablet 53bbd84258e04c76a2d1bf0af353444b from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:01.922013 7617 ts_tablet_manager.cc:1905] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:01.930323 7602 tablet_bootstrap.cc:492] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:01.931316 7602 ts_tablet_manager.cc:1397] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.095s user 0.007s sys 0.008s
I20250110 20:58:01.935222 7602 raft_consensus.cc:357] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.938634 7609 tablet_bootstrap.cc:654] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:01.944258 7333 tablet_service.cc:1467] Processing CreateTablet for tablet f3e994b6ed3e4692a4ba3cfcf3b72648 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.945181 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e994b6ed3e4692a4ba3cfcf3b72648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.941134 7189 tablet_service.cc:1467] Processing CreateTablet for tablet f3e994b6ed3e4692a4ba3cfcf3b72648 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.956390 7602 raft_consensus.cc:383] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:01.956652 7602 raft_consensus.cc:738] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:01.956898 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e994b6ed3e4692a4ba3cfcf3b72648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.957222 7602 consensus_queue.cc:260] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:01.950585 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 9f62814cd3544ba6a1bbe0550dd8734c (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.960744 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f62814cd3544ba6a1bbe0550dd8734c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.965817 7501 raft_consensus.cc:1270] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:01.966130 7210 raft_consensus.cc:1270] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:01.967397 7897 consensus_queue.cc:1035] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:01.968799 7897 consensus_queue.cc:1035] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:01.965992 7481 tablet_service.cc:1467] Processing CreateTablet for tablet f3e994b6ed3e4692a4ba3cfcf3b72648 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.981513 7602 ts_tablet_manager.cc:1428] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.050s user 0.005s sys 0.000s
I20250110 20:58:01.982221 7602 tablet_bootstrap.cc:492] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:01.976374 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3e994b6ed3e4692a4ba3cfcf3b72648. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.989516 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 9f62814cd3544ba6a1bbe0550dd8734c (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.990536 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f62814cd3544ba6a1bbe0550dd8734c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:01.991343 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 9f62814cd3544ba6a1bbe0550dd8734c (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:01.992246 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f62814cd3544ba6a1bbe0550dd8734c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.009740 6985 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:02.011319 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet 69f0a5ceae3a489b9b0552c550d3e80a from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:02.016505 7609 tablet_bootstrap.cc:492] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:02.016269 7932 tablet.cc:1721] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:02.016891 7609 ts_tablet_manager.cc:1397] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.131s user 0.009s sys 0.000s
I20250110 20:58:02.021200 7911 raft_consensus.cc:491] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:02.021631 7911 raft_consensus.cc:513] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.023459 7911 leader_election.cc:290] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:02.024938 7933 tablet.cc:1721] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:02.029812 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77661375951e456899d2f9052dc680e7" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:02.030124 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77661375951e456899d2f9052dc680e7" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:02.030417 7352 raft_consensus.cc:2463] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:02.031046 7210 raft_consensus.cc:2463] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:02.031486 7436 leader_election.cc:304] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.032267 7911 raft_consensus.cc:2798] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:02.032476 6992 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 53bbd84258e04c76a2d1bf0af353444b
I20250110 20:58:02.032570 7911 raft_consensus.cc:491] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:02.032867 7911 raft_consensus.cc:3054] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.035483 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bbd84258e04c76a2d1bf0af353444b with delete_type TABLET_DATA_DELETED (replaced by tablet 3d4511e84bd7406cafe8691e0216ad93) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.036087 7615 tablet_replica.cc:331] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:02.036739 7615 raft_consensus.cc:2238] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.037210 7615 raft_consensus.cc:2267] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.021741 7609 raft_consensus.cc:357] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.037909 7609 raft_consensus.cc:383] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.038182 7609 raft_consensus.cc:738] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.039808 7615 ts_tablet_manager.cc:1905] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.040771 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bbd84258e04c76a2d1bf0af353444b with delete_type TABLET_DATA_DELETED (replaced by tablet 3d4511e84bd7406cafe8691e0216ad93) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:02.041383 6992 catalog_manager.cc:6596] ReplaceTablet: tablet 53bbd84258e04c76a2d1bf0af353444b deleted and replaced by tablet 3d4511e84bd7406cafe8691e0216ad93
I20250110 20:58:02.042814 7330 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bbd84258e04c76a2d1bf0af353444b with delete_type TABLET_DATA_DELETED (replaced by tablet 3d4511e84bd7406cafe8691e0216ad93) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:02.047221 7931 tablet.cc:1721] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:02.048023 7609 consensus_queue.cc:260] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.050396 7609 ts_tablet_manager.cc:1428] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20250110 20:58:02.050714 6985 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:02.055335 7602 tablet_bootstrap.cc:654] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.055554 7911 raft_consensus.cc:513] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.057318 7911 leader_election.cc:290] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:02.060099 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77661375951e456899d2f9052dc680e7" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:02.060673 7352 raft_consensus.cc:3054] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.062463 7609 tablet_bootstrap.cc:492] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:02.067332 7352 raft_consensus.cc:2463] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:02.069535 7436 leader_election.cc:304] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.073895 7606 tablet_bootstrap.cc:492] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:02.074276 7606 ts_tablet_manager.cc:1397] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.228s user 0.004s sys 0.005s
I20250110 20:58:02.075268 7615 ts_tablet_manager.cc:1918] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:02.075323 7911 raft_consensus.cc:2798] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:02.075731 7911 raft_consensus.cc:695] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:58:02.078188 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77661375951e456899d2f9052dc680e7" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:02.079556 7210 raft_consensus.cc:3054] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.080181 7606 raft_consensus.cc:357] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.080801 7606 raft_consensus.cc:383] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.081063 7606 raft_consensus.cc:738] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.082834 7911 consensus_queue.cc:237] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.083500 7609 tablet_bootstrap.cc:654] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.083670 7616 ts_tablet_manager.cc:1918] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:02.084088 7616 log.cc:1198] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/6ad3b006f8364950bcf60d0b0888b819
I20250110 20:58:02.084738 7616 ts_tablet_manager.cc:1939] T 6ad3b006f8364950bcf60d0b0888b819 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:02.089933 7606 consensus_queue.cc:260] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.090875 7602 tablet_bootstrap.cc:492] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:02.091253 7602 ts_tablet_manager.cc:1397] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.109s user 0.008s sys 0.000s
I20250110 20:58:02.092182 6992 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:02.093554 7602 raft_consensus.cc:357] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.094177 7602 raft_consensus.cc:383] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.094411 7602 raft_consensus.cc:738] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.094933 7602 consensus_queue.cc:260] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.095906 7606 ts_tablet_manager.cc:1428] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20250110 20:58:02.098433 6985 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:02.102527 7606 tablet_bootstrap.cc:492] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:02.075608 7615 log.cc:1198] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/53bbd84258e04c76a2d1bf0af353444b
I20250110 20:58:02.105305 7615 ts_tablet_manager.cc:1939] T 53bbd84258e04c76a2d1bf0af353444b P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:02.110045 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 6ad3b006f8364950bcf60d0b0888b819 (table default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]) successfully deleted
I20250110 20:58:02.112866 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 53bbd84258e04c76a2d1bf0af353444b (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:58:02.115154 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bbd84258e04c76a2d1bf0af353444b with delete_type TABLET_DATA_DELETED (replaced by tablet 3d4511e84bd7406cafe8691e0216ad93) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:02.115871 7616 tablet_replica.cc:331] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:02.110352 7602 ts_tablet_manager.cc:1428] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20250110 20:58:02.121162 7602 tablet_bootstrap.cc:492] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:02.125876 7602 tablet_bootstrap.cc:654] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.128008 7609 tablet_bootstrap.cc:492] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:02.128420 7609 ts_tablet_manager.cc:1397] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.066s user 0.008s sys 0.000s
I20250110 20:58:02.130753 7606 tablet_bootstrap.cc:654] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.130679 7609 raft_consensus.cc:357] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.131340 7609 raft_consensus.cc:383] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.131592 7609 raft_consensus.cc:738] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.132181 7609 consensus_queue.cc:260] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.137621 7210 raft_consensus.cc:2463] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:02.138577 7616 raft_consensus.cc:2238] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:02.140861 7617 ts_tablet_manager.cc:1918] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:02.143743 7616 raft_consensus.cc:2267] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.152010 7616 ts_tablet_manager.cc:1905] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.152422 7617 log.cc:1198] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/40083e396e1b4a13a77e98ade14a0d04
I20250110 20:58:02.153604 7617 ts_tablet_manager.cc:1939] T 40083e396e1b4a13a77e98ade14a0d04 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:02.163558 7210 raft_consensus.cc:1270] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:02.164795 7926 consensus_queue.cc:1035] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:02.169745 7868 raft_consensus.cc:491] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:02.170153 7868 raft_consensus.cc:513] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.171972 7868 leader_election.cc:290] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.176210 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec62f901ffe64faea7d3716db20c487f" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:02.176757 7352 raft_consensus.cc:2463] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:02.180083 7145 leader_election.cc:304] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:02.182152 7606 tablet_bootstrap.cc:492] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:02.182520 7606 ts_tablet_manager.cc:1397] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.000s
I20250110 20:58:02.185256 7606 raft_consensus.cc:357] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.185910 7606 raft_consensus.cc:383] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.186143 7606 raft_consensus.cc:738] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.186746 7606 consensus_queue.cc:260] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.158128 7609 ts_tablet_manager.cc:1428] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20250110 20:58:02.194428 7609 tablet_bootstrap.cc:492] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:02.163398 7501 raft_consensus.cc:1270] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:02.177672 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec62f901ffe64faea7d3716db20c487f" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:02.196689 7500 raft_consensus.cc:2463] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:02.201429 7609 tablet_bootstrap.cc:654] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.203524 7926 consensus_queue.cc:1035] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:02.209079 7602 tablet_bootstrap.cc:492] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:02.209484 7602 ts_tablet_manager.cc:1397] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.088s user 0.026s sys 0.032s
I20250110 20:58:02.211102 7868 raft_consensus.cc:2798] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:02.211489 7868 raft_consensus.cc:491] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:02.211912 7868 raft_consensus.cc:3054] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.212097 7602 raft_consensus.cc:357] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.212872 7602 raft_consensus.cc:383] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.213119 7602 raft_consensus.cc:738] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.213703 7602 consensus_queue.cc:260] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.216610 7606 ts_tablet_manager.cc:1428] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20250110 20:58:02.218145 7609 tablet_bootstrap.cc:492] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:02.218556 7609 ts_tablet_manager.cc:1397] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.000s
I20250110 20:58:02.220659 7609 raft_consensus.cc:357] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.221372 7609 raft_consensus.cc:383] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.221697 7609 raft_consensus.cc:738] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.222393 7609 consensus_queue.cc:260] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.222982 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4511e84bd7406cafe8691e0216ad93 (DEFAULT_TABLE table=default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.223997 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4511e84bd7406cafe8691e0216ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.224428 7609 ts_tablet_manager.cc:1428] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250110 20:58:02.225150 7609 tablet_bootstrap.cc:492] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:02.227952 6985 catalog_manager.cc:5526] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:58:02.230322 7868 raft_consensus.cc:513] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.233801 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4511e84bd7406cafe8691e0216ad93 (DEFAULT_TABLE table=default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.234717 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4511e84bd7406cafe8691e0216ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.254952 7609 tablet_bootstrap.cc:654] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.236882 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4511e84bd7406cafe8691e0216ad93 (DEFAULT_TABLE table=default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.238873 7950 raft_consensus.cc:491] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:02.250386 7602 ts_tablet_manager.cc:1428] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.041s user 0.008s sys 0.001s
I20250110 20:58:02.256424 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d4511e84bd7406cafe8691e0216ad93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.256062 7950 raft_consensus.cc:513] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.257736 7602 tablet_bootstrap.cc:492] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:02.257988 7950 leader_election.cc:290] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.263254 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e994b6ed3e4692a4ba3cfcf3b72648" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:02.263939 7602 tablet_bootstrap.cc:654] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.262161 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e994b6ed3e4692a4ba3cfcf3b72648" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:02.265887 7606 tablet_bootstrap.cc:492] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
W20250110 20:58:02.264971 7145 leader_election.cc:343] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:58:02.266646 7145 leader_election.cc:343] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:02.267097 7145 leader_election.cc:304] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:02.274513 7950 raft_consensus.cc:2743] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:02.291786 7602 tablet_bootstrap.cc:492] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:02.292291 7602 ts_tablet_manager.cc:1397] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.035s user 0.017s sys 0.001s
I20250110 20:58:02.294489 7602 raft_consensus.cc:357] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.296594 7609 tablet_bootstrap.cc:492] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:02.297013 7609 ts_tablet_manager.cc:1397] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.005s
I20250110 20:58:02.299145 7609 raft_consensus.cc:357] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.300038 7609 raft_consensus.cc:383] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.300328 7609 raft_consensus.cc:738] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.301533 7602 raft_consensus.cc:383] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.301878 7602 raft_consensus.cc:738] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.300976 7609 consensus_queue.cc:260] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.302373 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec62f901ffe64faea7d3716db20c487f" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:02.302902 7501 raft_consensus.cc:3054] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.306574 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec62f901ffe64faea7d3716db20c487f" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:02.307170 7352 raft_consensus.cc:3054] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.307780 7749 raft_consensus.cc:491] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:02.308157 7749 raft_consensus.cc:513] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.309837 7749 leader_election.cc:290] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:02.311246 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e994b6ed3e4692a4ba3cfcf3b72648" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:02.311815 7209 raft_consensus.cc:2463] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:02.312887 7435 leader_election.cc:304] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.313684 7749 raft_consensus.cc:2798] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:02.313987 7749 raft_consensus.cc:491] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:02.314260 7749 raft_consensus.cc:3054] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.315479 7501 raft_consensus.cc:2463] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:02.316427 7145 leader_election.cc:304] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.317093 7950 raft_consensus.cc:2798] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:02.317641 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e994b6ed3e4692a4ba3cfcf3b72648" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
W20250110 20:58:02.319548 7436 leader_election.cc:343] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:02.322640 7933 tablet.cc:1721] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:02.323308 7749 raft_consensus.cc:513] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.325717 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e994b6ed3e4692a4ba3cfcf3b72648" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:02.326272 7209 raft_consensus.cc:3054] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.327319 7606 tablet_bootstrap.cc:654] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.333017 7209 raft_consensus.cc:2463] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:02.302920 7602 consensus_queue.cc:260] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.334668 7435 leader_election.cc:304] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.335465 7914 raft_consensus.cc:2798] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:02.335848 7914 raft_consensus.cc:695] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:58:02.336577 7914 consensus_queue.cc:237] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.307463 7609 ts_tablet_manager.cc:1428] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.010s user 0.002s sys 0.003s
I20250110 20:58:02.345093 7609 tablet_bootstrap.cc:492] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:02.345736 6994 catalog_manager.cc:2686] deleted HMS catalog entry for table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]
I20250110 20:58:02.350581 7749 leader_election.cc:290] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:02.352368 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3e994b6ed3e4692a4ba3cfcf3b72648" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
W20250110 20:58:02.353901 7436 leader_election.cc:343] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:02.355569 7602 ts_tablet_manager.cc:1428] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.063s user 0.006s sys 0.001s
I20250110 20:58:02.356854 7931 tablet.cc:1721] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:02.364143 7616 ts_tablet_manager.cc:1918] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:02.364549 7616 log.cc:1198] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/53bbd84258e04c76a2d1bf0af353444b
I20250110 20:58:02.366003 7616 ts_tablet_manager.cc:1939] T 53bbd84258e04c76a2d1bf0af353444b P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:02.367151 7352 raft_consensus.cc:2463] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:02.367924 7609 tablet_bootstrap.cc:654] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.383955 6992 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:02.389353 7602 tablet_bootstrap.cc:492] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:02.395185 7602 tablet_bootstrap.cc:654] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.395120 7932 tablet.cc:1721] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:02.402324 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 40083e396e1b4a13a77e98ade14a0d04 (table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]) successfully deleted
I20250110 20:58:02.301472 7868 leader_election.cc:290] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.423213 7617 tablet_replica.cc:331] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:02.424098 7617 raft_consensus.cc:2238] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.424690 7617 raft_consensus.cc:2267] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.431205 7617 ts_tablet_manager.cc:1905] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.431599 7954 raft_consensus.cc:491] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:02.432049 7954 raft_consensus.cc:513] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.432840 6985 catalog_manager.cc:6071] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376] alter complete (version 1)
I20250110 20:58:02.437108 7954 leader_election.cc:290] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.438279 6985 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:02.438783 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c67636cbc4fe98742a00b970f3701" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:02.439401 7500 raft_consensus.cc:2463] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:02.440589 7145 leader_election.cc:304] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.441483 7954 raft_consensus.cc:2798] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:02.442210 7954 raft_consensus.cc:491] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:02.442566 7954 raft_consensus.cc:3054] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.443949 7950 raft_consensus.cc:695] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:58:02.444794 7950 consensus_queue.cc:237] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.448506 7609 tablet_bootstrap.cc:492] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:02.448575 7602 tablet_bootstrap.cc:492] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:02.448971 7609 ts_tablet_manager.cc:1397] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.104s user 0.023s sys 0.003s
I20250110 20:58:02.449012 7602 ts_tablet_manager.cc:1397] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.060s user 0.008s sys 0.002s
I20250110 20:58:02.448688 7954 raft_consensus.cc:513] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.446564 7049 catalog_manager.cc:2710] Deleting table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505] in response to Hive Metastore notification log event 19
I20250110 20:58:02.451215 7609 raft_consensus.cc:357] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.451812 7609 raft_consensus.cc:383] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.451841 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c67636cbc4fe98742a00b970f3701" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:02.452139 7609 raft_consensus.cc:738] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.452420 7500 raft_consensus.cc:3054] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.452598 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 53bbd84258e04c76a2d1bf0af353444b (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:58:02.452718 7609 consensus_queue.cc:260] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.458640 7500 raft_consensus.cc:2463] T 721c67636cbc4fe98742a00b970f3701 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:02.433791 7606 tablet_bootstrap.cc:492] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:02.439710 7352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c67636cbc4fe98742a00b970f3701" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:02.466034 7352 raft_consensus.cc:2463] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:02.450968 7954 leader_election.cc:290] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.451670 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "721c67636cbc4fe98742a00b970f3701" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:02.467115 7609 ts_tablet_manager.cc:1428] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20250110 20:58:02.467250 7353 raft_consensus.cc:3054] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.469512 7145 leader_election.cc:304] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.470228 7868 raft_consensus.cc:2798] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:02.470638 7868 raft_consensus.cc:695] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:58:02.471343 7868 consensus_queue.cc:237] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.473649 7606 ts_tablet_manager.cc:1397] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.208s user 0.001s sys 0.008s
I20250110 20:58:02.480571 7353 raft_consensus.cc:2463] T 721c67636cbc4fe98742a00b970f3701 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:02.481599 7606 raft_consensus.cc:357] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.482321 7606 raft_consensus.cc:383] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.482600 7606 raft_consensus.cc:738] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.483273 7606 consensus_queue.cc:260] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.467646 7954 raft_consensus.cc:491] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:02.489246 7954 raft_consensus.cc:513] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.493852 7606 ts_tablet_manager.cc:1428] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.020s user 0.004s sys 0.002s
I20250110 20:58:02.498060 7602 raft_consensus.cc:357] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.500980 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f62814cd3544ba6a1bbe0550dd8734c" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:02.501593 7500 raft_consensus.cc:2463] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:02.505838 7049 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet e73e60dfdab34bee95fd3bfc2fede7be
I20250110 20:58:02.511788 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet e73e60dfdab34bee95fd3bfc2fede7be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:58:02 UTC) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:02.512530 7616 tablet_replica.cc:331] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:02.513263 7616 raft_consensus.cc:2238] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:02.514247 7616 raft_consensus.cc:2267] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.514346 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet 9f62814cd3544ba6a1bbe0550dd8734c from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:02.515702 7049 catalog_manager.cc:2710] Deleting table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f] in response to Hive Metastore notification log event 20
I20250110 20:58:02.516078 7616 ts_tablet_manager.cc:1905] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.497359 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f62814cd3544ba6a1bbe0550dd8734c" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:02.511186 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet e73e60dfdab34bee95fd3bfc2fede7be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:58:02 UTC) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:02.525195 7353 raft_consensus.cc:2463] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:02.531634 7606 tablet_bootstrap.cc:492] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:02.502779 7145 leader_election.cc:304] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:02.503288 7954 leader_election.cc:290] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.538350 7868 raft_consensus.cc:2798] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:02.538656 7868 raft_consensus.cc:491] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:02.538841 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet e73e60dfdab34bee95fd3bfc2fede7be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:58:02 UTC) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.538980 7868 raft_consensus.cc:3054] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.539319 7602 raft_consensus.cc:383] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.539469 7615 tablet_replica.cc:331] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:02.539547 7602 raft_consensus.cc:738] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.540246 7615 raft_consensus.cc:2238] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.540660 7615 raft_consensus.cc:2267] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.542269 7615 ts_tablet_manager.cc:1905] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.554275 7868 raft_consensus.cc:513] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.556711 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f62814cd3544ba6a1bbe0550dd8734c" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:02.557343 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f62814cd3544ba6a1bbe0550dd8734c" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:02.557713 7353 raft_consensus.cc:3054] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.557888 7500 raft_consensus.cc:3054] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.558051 6985 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 9f62814cd3544ba6a1bbe0550dd8734c
I20250110 20:58:02.559549 6985 catalog_manager.cc:6596] ReplaceTablet: tablet 9f62814cd3544ba6a1bbe0550dd8734c deleted and replaced by tablet fb8d5bf6a9694240b10e1b1c178b528b
I20250110 20:58:02.562384 7049 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 164f26db984343a390f253aba1219f4a
I20250110 20:58:02.563019 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f62814cd3544ba6a1bbe0550dd8734c with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d5bf6a9694240b10e1b1c178b528b) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:02.563373 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f62814cd3544ba6a1bbe0550dd8734c with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d5bf6a9694240b10e1b1c178b528b) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.563688 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f26db984343a390f253aba1219f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:58:02 UTC) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.564100 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f62814cd3544ba6a1bbe0550dd8734c with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d5bf6a9694240b10e1b1c178b528b) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:02.540262 7602 consensus_queue.cc:260] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.565809 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f26db984343a390f253aba1219f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:58:02 UTC) from {username='slave'} at 127.0.0.1:40394
W20250110 20:58:02.566269 6989 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_cee5a6c92d284bf2ab6f7eb541bbb600 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:58:02.567178 7602 ts_tablet_manager.cc:1428] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.118s user 0.001s sys 0.003s
I20250110 20:58:02.569144 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 164f26db984343a390f253aba1219f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-01-10 20:58:02 UTC) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:02.569579 7353 raft_consensus.cc:2463] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:02.571776 7145 leader_election.cc:304] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:02.571931 7616 ts_tablet_manager.cc:1918] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:02.572254 7616 log.cc:1198] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/e73e60dfdab34bee95fd3bfc2fede7be
I20250110 20:58:02.572858 7616 ts_tablet_manager.cc:1939] T e73e60dfdab34bee95fd3bfc2fede7be P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:02.572942 7954 raft_consensus.cc:2798] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:02.573365 7954 raft_consensus.cc:695] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:58:02.573772 7500 raft_consensus.cc:2463] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
W20250110 20:58:02.574431 6986 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4dac0491728e4031866a732faeb6d42c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:58:02.574111 7954 consensus_queue.cc:237] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.575158 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet e73e60dfdab34bee95fd3bfc2fede7be (table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]) successfully deleted
I20250110 20:58:02.575186 7868 leader_election.cc:290] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.579815 6988 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33774 (request call id 60) took 4013 ms. Trace:
I20250110 20:58:02.580128 6988 rpcz_store.cc:276] 0110 20:57:58.566490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:58.611851 (+ 45361us) service_pool.cc:225] Handling call
0110 20:57:59.681428 (+1069577us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:58:02.577109 (+2895681us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:58:02.578928 (+ 1819us) catalog_manager.cc:3627] Apply alter schema
0110 20:58:02.579751 (+ 823us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":908877,"Hive Metastore.run_cpu_time_us":1671,"Hive Metastore.run_wall_time_us":493367}
I20250110 20:58:02.584764 6990 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33774 (request call id 66) took 3462 ms. Trace:
I20250110 20:58:02.585040 6990 rpcz_store.cc:276] 0110 20:57:59.122363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:59.122498 (+ 135us) service_pool.cc:225] Handling call
0110 20:57:59.157899 (+ 35401us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b5400003390. stack: 00007f728d56d853 00007f728d56d750 00007f728d0e2584 00000000004fcd24 00000000004fccb9 00007f728d3e32d1 00007f728d562d60 00007f72961762ba 00007f729628f9c2 00007f7293ffc644 00007f7293ffc5d2 00007f7293ffc501 00007f7293ffc47c 00007f7293ffb732 00007f72922b5dfc 00007f72922b5236
0110 20:58:00.621574 (+1463675us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:58:02.583293 (+1961719us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:58:02.584050 (+ 757us) catalog_manager.cc:3627] Apply alter schema
0110 20:58:02.584711 (+ 661us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":462414,"Hive Metastore.run_cpu_time_us":1659,"Hive Metastore.run_wall_time_us":337783,"spinlock_wait_cycles":7995136}
I20250110 20:58:02.585001 6992 catalog_manager.cc:5526] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:02.586150 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet ec62f901ffe64faea7d3716db20c487f from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:02.587608 7616 tablet_replica.cc:331] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:02.588326 7616 raft_consensus.cc:2238] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.590219 7616 raft_consensus.cc:2267] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.592156 7616 ts_tablet_manager.cc:1905] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.592135 7615 ts_tablet_manager.cc:1918] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:02.592474 7615 log.cc:1198] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/e73e60dfdab34bee95fd3bfc2fede7be
I20250110 20:58:02.593556 7615 ts_tablet_manager.cc:1939] T e73e60dfdab34bee95fd3bfc2fede7be P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:02.595645 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet e73e60dfdab34bee95fd3bfc2fede7be (table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]) successfully deleted
I20250110 20:58:02.609673 7615 tablet_replica.cc:331] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:02.611912 7615 raft_consensus.cc:2238] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:02.613075 7615 pending_rounds.cc:62] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:58:02.613294 7615 pending_rounds.cc:69] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878827831791616 op_type: NO_OP noop_request { }
I20250110 20:58:02.613649 7615 raft_consensus.cc:2883] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:02.613962 7615 raft_consensus.cc:2267] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.629146 7606 tablet_bootstrap.cc:654] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.632073 7616 ts_tablet_manager.cc:1918] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:02.632407 7616 log.cc:1198] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/9f62814cd3544ba6a1bbe0550dd8734c
I20250110 20:58:02.634574 6992 catalog_manager.cc:5526] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:02.633059 7616 ts_tablet_manager.cc:1939] T 9f62814cd3544ba6a1bbe0550dd8734c P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:02.643565 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 9f62814cd3544ba6a1bbe0550dd8734c (table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]) successfully deleted
I20250110 20:58:02.645210 7616 tablet_replica.cc:331] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:02.645892 7616 raft_consensus.cc:2238] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
W20250110 20:58:02.645843 7962 consensus_queue.cc:1159] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "244d9ba396f549f2a5015792b6c5fff8" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250110 20:58:02.650924 7615 ts_tablet_manager.cc:1905] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.657852 7616 raft_consensus.cc:2267] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.659475 7616 ts_tablet_manager.cc:1905] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.666538 7615 ts_tablet_manager.cc:1918] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:02.666839 7615 log.cc:1198] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/9f62814cd3544ba6a1bbe0550dd8734c
I20250110 20:58:02.667451 7615 ts_tablet_manager.cc:1939] T 9f62814cd3544ba6a1bbe0550dd8734c P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:02.669780 7615 tablet_replica.cc:331] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:02.669646 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 9f62814cd3544ba6a1bbe0550dd8734c (table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]) successfully deleted
I20250110 20:58:02.670399 7615 raft_consensus.cc:2238] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.670841 7615 raft_consensus.cc:2267] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.672479 7615 ts_tablet_manager.cc:1905] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.682678 7617 ts_tablet_manager.cc:1918] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:02.694980 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet ec62f901ffe64faea7d3716db20c487f
I20250110 20:58:02.696110 6990 catalog_manager.cc:6596] ReplaceTablet: tablet ec62f901ffe64faea7d3716db20c487f deleted and replaced by tablet 833cc8d477374f51a67422cbe051254e
I20250110 20:58:02.688083 7606 tablet_bootstrap.cc:492] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:02.696712 7606 ts_tablet_manager.cc:1397] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.165s user 0.008s sys 0.002s
I20250110 20:58:02.697256 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet ec62f901ffe64faea7d3716db20c487f with delete_type TABLET_DATA_DELETED (replaced by tablet 833cc8d477374f51a67422cbe051254e) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:02.697803 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet ec62f901ffe64faea7d3716db20c487f with delete_type TABLET_DATA_DELETED (replaced by tablet 833cc8d477374f51a67422cbe051254e) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:02.699774 7606 raft_consensus.cc:357] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.700636 7606 raft_consensus.cc:383] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.701970 7606 raft_consensus.cc:738] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.704066 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet ec62f901ffe64faea7d3716db20c487f with delete_type TABLET_DATA_DELETED (replaced by tablet 833cc8d477374f51a67422cbe051254e) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.705469 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f62814cd3544ba6a1bbe0550dd8734c with delete_type TABLET_DATA_DELETED (replaced by tablet fb8d5bf6a9694240b10e1b1c178b528b) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.701012 7617 log.cc:1198] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/53bbd84258e04c76a2d1bf0af353444b
I20250110 20:58:02.708779 7606 consensus_queue.cc:260] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.707942 7617 ts_tablet_manager.cc:1939] T 53bbd84258e04c76a2d1bf0af353444b P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:02.725927 7606 ts_tablet_manager.cc:1428] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20250110 20:58:02.729364 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 53bbd84258e04c76a2d1bf0af353444b (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
W20250110 20:58:02.729995 6980 catalog_manager.cc:4839] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): delete failed for tablet 53bbd84258e04c76a2d1bf0af353444b because the tablet was not found. No further retry: Not found: Tablet not found: 53bbd84258e04c76a2d1bf0af353444b
I20250110 20:58:02.735388 7615 ts_tablet_manager.cc:1918] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:02.735738 7615 log.cc:1198] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/164f26db984343a390f253aba1219f4a
I20250110 20:58:02.736382 7615 ts_tablet_manager.cc:1939] T 164f26db984343a390f253aba1219f4a P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:02.736794 7606 tablet_bootstrap.cc:492] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:02.738472 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:58:02.738750 7615 tablet_replica.cc:331] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:02.739387 7615 raft_consensus.cc:2238] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:02.739984 7615 pending_rounds.cc:62] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:58:02.740190 7615 pending_rounds.cc:69] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878827307073536 op_type: NO_OP noop_request { }
I20250110 20:58:02.727797 7617 tablet_replica.cc:331] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:02.740509 7615 raft_consensus.cc:2883] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:02.740784 7615 raft_consensus.cc:2267] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.742919 7615 ts_tablet_manager.cc:1905] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.749841 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 833cc8d477374f51a67422cbe051254e (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.750728 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 833cc8d477374f51a67422cbe051254e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.750352 6985 catalog_manager.cc:5526] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:58:02.751725 6985 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:02.753649 7189 tablet_service.cc:1467] Processing CreateTablet for tablet fb8d5bf6a9694240b10e1b1c178b528b (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.754099 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 833cc8d477374f51a67422cbe051254e (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.754238 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 833cc8d477374f51a67422cbe051254e (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.754504 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8d5bf6a9694240b10e1b1c178b528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.755062 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 833cc8d477374f51a67422cbe051254e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.755108 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 833cc8d477374f51a67422cbe051254e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.754233 7477 tablet_service.cc:1467] Processing CreateTablet for tablet fb8d5bf6a9694240b10e1b1c178b528b (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.757160 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8d5bf6a9694240b10e1b1c178b528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.758606 7332 tablet_service.cc:1467] Processing CreateTablet for tablet fb8d5bf6a9694240b10e1b1c178b528b (DEFAULT_TABLE table=default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.759498 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb8d5bf6a9694240b10e1b1c178b528b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.765581 7616 ts_tablet_manager.cc:1918] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:02.765915 7616 log.cc:1198] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/164f26db984343a390f253aba1219f4a
I20250110 20:58:02.766566 7616 ts_tablet_manager.cc:1939] T 164f26db984343a390f253aba1219f4a P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:02.768692 7616 tablet_replica.cc:331] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:02.770355 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 77661375951e456899d2f9052dc680e7 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:02.773762 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:58:02.775704 7617 raft_consensus.cc:2238] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.776805 7617 raft_consensus.cc:2267] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.778514 7617 ts_tablet_manager.cc:1905] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.783207 7609 tablet_bootstrap.cc:492] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:02.785799 7602 tablet_bootstrap.cc:492] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:02.788208 7609 tablet_bootstrap.cc:654] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.791352 7602 tablet_bootstrap.cc:654] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.793663 7615 ts_tablet_manager.cc:1918] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:02.794004 7615 log.cc:1198] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ec62f901ffe64faea7d3716db20c487f
I20250110 20:58:02.769352 7616 raft_consensus.cc:2238] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.795580 7616 raft_consensus.cc:2267] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.801534 6988 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
W20250110 20:58:02.803085 6988 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3d4511e84bd7406cafe8691e0216ad93 consensus state has no leader
I20250110 20:58:02.804999 7616 ts_tablet_manager.cc:1905] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.806484 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 77661375951e456899d2f9052dc680e7
I20250110 20:58:02.808422 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 77661375951e456899d2f9052dc680e7 deleted and replaced by tablet 43834def62374039abcbcab349edfe2e
I20250110 20:58:02.811923 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 77661375951e456899d2f9052dc680e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 43834def62374039abcbcab349edfe2e) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:02.812757 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 77661375951e456899d2f9052dc680e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 43834def62374039abcbcab349edfe2e) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:02.821748 6992 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_59954cb851e943ff9f10bca64b86d8b8" } new_table_name: "default.Table_c3ed0987ec6e41d39ddc47cba082a635" modify_external_catalogs: true
I20250110 20:58:02.821522 7606 tablet_bootstrap.cc:654] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:02.823518 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:02.823714 6988 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_ba5d6def0428488589bda04057a79fab" } new_table_name: "default.Table_58cf4fbf7d1a441ebe2a9611029cf17b" modify_external_catalogs: true
I20250110 20:58:02.807986 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 77661375951e456899d2f9052dc680e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 43834def62374039abcbcab349edfe2e) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.819442 7602 tablet_bootstrap.cc:492] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:02.826910 7602 ts_tablet_manager.cc:1397] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.004s
I20250110 20:58:02.827663 7609 tablet_bootstrap.cc:492] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:02.828059 7609 ts_tablet_manager.cc:1397] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.008s
I20250110 20:58:02.829267 7602 raft_consensus.cc:357] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.829978 7602 raft_consensus.cc:383] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.830727 7609 raft_consensus.cc:357] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.831357 7609 raft_consensus.cc:383] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.794626 7615 ts_tablet_manager.cc:1939] T ec62f901ffe64faea7d3716db20c487f P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:02.831409 7602 raft_consensus.cc:738] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.833539 7602 consensus_queue.cc:260] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.834267 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet ec62f901ffe64faea7d3716db20c487f (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]) successfully deleted
W20250110 20:58:02.836473 6979 catalog_manager.cc:4839] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet 9f62814cd3544ba6a1bbe0550dd8734c because the tablet was not found. No further retry: Not found: Tablet not found: 9f62814cd3544ba6a1bbe0550dd8734c
I20250110 20:58:02.837461 7609 raft_consensus.cc:738] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.838101 7609 consensus_queue.cc:260] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.837230 7615 tablet_replica.cc:331] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:02.839358 7615 raft_consensus.cc:2238] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.846402 7615 raft_consensus.cc:2267] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.847199 7602 ts_tablet_manager.cc:1428] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.020s user 0.005s sys 0.000s
I20250110 20:58:02.854100 7602 tablet_bootstrap.cc:492] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:02.867082 7609 ts_tablet_manager.cc:1428] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250110 20:58:02.867902 7609 tablet_bootstrap.cc:492] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:02.873672 7616 ts_tablet_manager.cc:1918] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:02.847975 7615 ts_tablet_manager.cc:1905] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:58:02.884892 6980 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3d4511e84bd7406cafe8691e0216ad93 consensus state has no leader
I20250110 20:58:02.887370 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 43834def62374039abcbcab349edfe2e (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.888291 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43834def62374039abcbcab349edfe2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:58:02.890161 6985 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:02.907459 7615 ts_tablet_manager.cc:1918] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:02.907778 7615 log.cc:1198] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/77661375951e456899d2f9052dc680e7
I20250110 20:58:02.908340 7615 ts_tablet_manager.cc:1939] T 77661375951e456899d2f9052dc680e7 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:02.909399 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 77661375951e456899d2f9052dc680e7 (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:02.912986 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet ac84d3932d5e421fa0b641419eeebe6a from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:02.913558 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 43834def62374039abcbcab349edfe2e (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.914505 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43834def62374039abcbcab349edfe2e. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:58:02.924782 6980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20250110 20:58:02.924571 7602 tablet_bootstrap.cc:654] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:02.928300 6980 consensus_peers.cc:487] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08 -> Peer f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203): Couldn't send request to peer f9bb61bdf30044a1b482d6a66e29435f. Status: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20250110 20:58:02.931629 7606 tablet_bootstrap.cc:492] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:02.898626 7609 tablet_bootstrap.cc:654] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:02.932039 7606 ts_tablet_manager.cc:1397] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.196s user 0.010s sys 0.033s
I20250110 20:58:02.932029 7954 raft_consensus.cc:491] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:02.932457 7954 raft_consensus.cc:513] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.934360 7954 leader_election.cc:290] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:02.924268 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 43834def62374039abcbcab349edfe2e (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:02.935708 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4511e84bd7406cafe8691e0216ad93" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:02.936048 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43834def62374039abcbcab349edfe2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:02.935894 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4511e84bd7406cafe8691e0216ad93" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:02.936465 7500 raft_consensus.cc:2463] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:02.938400 7616 log.cc:1198] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ec62f901ffe64faea7d3716db20c487f
I20250110 20:58:02.941752 7145 leader_election.cc:304] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
W20250110 20:58:02.942500 7145 leader_election.cc:343] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:02.949599 7954 raft_consensus.cc:2798] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:02.949944 7954 raft_consensus.cc:491] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:02.950265 7954 raft_consensus.cc:3054] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.949749 7616 ts_tablet_manager.cc:1939] T ec62f901ffe64faea7d3716db20c487f P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:02.950277 7606 raft_consensus.cc:357] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.955170 7616 tablet_replica.cc:331] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:02.955878 7616 raft_consensus.cc:2238] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:02.956553 7616 pending_rounds.cc:62] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Trying to abort 1 pending ops.
I20250110 20:58:02.956812 7616 pending_rounds.cc:69] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878825886629888 op_type: NO_OP noop_request { }
I20250110 20:58:02.957219 7616 raft_consensus.cc:2883] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:02.957587 7616 raft_consensus.cc:2267] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.958030 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet ec62f901ffe64faea7d3716db20c487f (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]) successfully deleted
I20250110 20:58:02.959915 7616 ts_tablet_manager.cc:1905] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.965658 7602 tablet_bootstrap.cc:492] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:02.965827 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet ac84d3932d5e421fa0b641419eeebe6a
I20250110 20:58:02.966056 7602 ts_tablet_manager.cc:1397] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.002s
I20250110 20:58:02.968273 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet ac84d3932d5e421fa0b641419eeebe6a with delete_type TABLET_DATA_DELETED (replaced by tablet a580b2f85d854ff9979333b409bd6a77) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:02.968433 6990 catalog_manager.cc:6596] ReplaceTablet: tablet ac84d3932d5e421fa0b641419eeebe6a deleted and replaced by tablet a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:02.968590 7602 raft_consensus.cc:357] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.973687 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet ac84d3932d5e421fa0b641419eeebe6a with delete_type TABLET_DATA_DELETED (replaced by tablet a580b2f85d854ff9979333b409bd6a77) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:02.975431 6993 catalog_manager.cc:2149] created HMS catalog entry for table [id=150bdbba49cc4d7b80896013ce4acadd]
I20250110 20:58:02.977162 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet ac84d3932d5e421fa0b641419eeebe6a with delete_type TABLET_DATA_DELETED (replaced by tablet a580b2f85d854ff9979333b409bd6a77) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:02.977840 7615 tablet_replica.cc:331] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:02.978490 7615 raft_consensus.cc:2238] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:02.978946 7615 raft_consensus.cc:2267] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:02.957115 7954 raft_consensus.cc:513] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.950930 7606 raft_consensus.cc:383] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.984035 7615 ts_tablet_manager.cc:1905] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:02.985275 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4511e84bd7406cafe8691e0216ad93" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:02.986914 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4511e84bd7406cafe8691e0216ad93" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:02.969182 7602 raft_consensus.cc:383] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:02.987474 7500 raft_consensus.cc:3054] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:02.987561 7602 raft_consensus.cc:738] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:02.988240 7602 consensus_queue.cc:260] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.989876 6993 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 77) took 2765 ms. Trace:
I20250110 20:58:02.993582 7500 raft_consensus.cc:2463] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:02.983945 7606 raft_consensus.cc:738] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
W20250110 20:58:02.997294 7287 consensus_peers.cc:487] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Couldn't send request to peer 244d9ba396f549f2a5015792b6c5fff8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:58:02.997646 7606 consensus_queue.cc:260] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:02.998461 7353 raft_consensus.cc:3054] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.002202 7617 ts_tablet_manager.cc:1918] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:02.990197 6993 rpcz_store.cc:276] 0110 20:58:00.224388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:00.240048 (+ 15660us) service_pool.cc:225] Handling call
0110 20:58:01.686260 (+1446212us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:58:01.689121 (+ 2861us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:58:02.975386 (+1286265us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:02.989204 (+ 13818us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:02.989543 (+ 339us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:02.989818 (+ 275us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:58:02.976990 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:58:02.977356 (+ 366us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:58:02.977379 (+ 23us) write_op.cc:435] Acquired schema lock
0110 20:58:02.977391 (+ 12us) tablet.cc:659] Decoding operations
0110 20:58:02.977669 (+ 278us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:58:02.977708 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0110 20:58:02.977728 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:58:02.977904 (+ 176us) tablet.cc:698] Row locks acquired
0110 20:58:02.977918 (+ 14us) write_op.cc:260] PREPARE: finished
0110 20:58:02.978024 (+ 106us) write_op.cc:270] Start()
0110 20:58:02.978099 (+ 75us) write_op.cc:276] Timestamp: P: 1736542682977960 usec, L: 0
0110 20:58:02.978112 (+ 13us) op_driver.cc:348] REPLICATION: starting
0110 20:58:02.978585 (+ 473us) log.cc:844] Serialized 433 byte log entry
0110 20:58:02.987507 (+ 8922us) op_driver.cc:464] REPLICATION: finished
0110 20:58:02.987881 (+ 374us) write_op.cc:301] APPLY: starting
0110 20:58:02.987932 (+ 51us) tablet.cc:1364] starting BulkCheckPresence
0110 20:58:02.988028 (+ 96us) tablet.cc:1367] finished BulkCheckPresence
0110 20:58:02.988112 (+ 84us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:58:02.988474 (+ 362us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:58:02.988603 (+ 129us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:58:02.988643 (+ 40us) write_op.cc:312] APPLY: finished
0110 20:58:02.988802 (+ 159us) log.cc:844] Serialized 29 byte log entry
0110 20:58:02.988937 (+ 135us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:02.989005 (+ 68us) write_op.cc:454] Released schema lock
0110 20:58:02.989033 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0110 20:58:02.989246 (+ 213us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:02.989269 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":751860,"Hive Metastore.run_cpu_time_us":549,"Hive Metastore.run_wall_time_us":533247,"child_traces":[["op",{"apply.queue_time_us":237,"apply.run_cpu_time_us":1329,"apply.run_wall_time_us":1329,"num_ops":2,"prepare.queue_time_us":98,"prepare.run_cpu_time_us":1910,"prepare.run_wall_time_us":3654,"raft.queue_time_us":118,"raft.run_cpu_time_us":538,"raft.run_wall_time_us":539,"replication_time_us":9319}]]}
I20250110 20:58:03.002553 7617 log.cc:1198] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/e73e60dfdab34bee95fd3bfc2fede7be
I20250110 20:58:03.003219 7617 ts_tablet_manager.cc:1939] T e73e60dfdab34bee95fd3bfc2fede7be P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:03.004545 7353 raft_consensus.cc:2463] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:02.994378 7145 leader_election.cc:304] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:03.005360 7868 raft_consensus.cc:2798] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:03.005805 7868 raft_consensus.cc:695] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:58:03.006525 7868 consensus_queue.cc:237] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.015534 7602 ts_tablet_manager.cc:1428] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.049s user 0.005s sys 0.001s
I20250110 20:58:03.020578 6993 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:03.017848 7602 tablet_bootstrap.cc:492] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:03.023599 6990 catalog_manager.cc:5526] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:03.024776 6990 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:03.030151 6985 master_service.cc:946] ReplaceTablet: received request to replace tablet f25437547b454eafbcf56e2b6c3b348a from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:03.043375 6985 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet f25437547b454eafbcf56e2b6c3b348a
I20250110 20:58:02.995039 7954 leader_election.cc:290] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.044696 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet f25437547b454eafbcf56e2b6c3b348a with delete_type TABLET_DATA_DELETED (replaced by tablet 12429ac2edaf49b9bdddb611e97ef1c3) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:03.044945 6985 catalog_manager.cc:6596] ReplaceTablet: tablet f25437547b454eafbcf56e2b6c3b348a deleted and replaced by tablet 12429ac2edaf49b9bdddb611e97ef1c3
I20250110 20:58:03.047358 7331 tablet_service.cc:1514] Processing DeleteTablet for tablet f25437547b454eafbcf56e2b6c3b348a with delete_type TABLET_DATA_DELETED (replaced by tablet 12429ac2edaf49b9bdddb611e97ef1c3) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:03.057313 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet f25437547b454eafbcf56e2b6c3b348a with delete_type TABLET_DATA_DELETED (replaced by tablet 12429ac2edaf49b9bdddb611e97ef1c3) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:03.060526 7606 ts_tablet_manager.cc:1428] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.128s user 0.003s sys 0.003s
I20250110 20:58:03.062721 7609 tablet_bootstrap.cc:492] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:03.063164 7609 ts_tablet_manager.cc:1397] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.195s user 0.011s sys 0.013s
I20250110 20:58:03.065338 7609 raft_consensus.cc:357] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.066114 7609 raft_consensus.cc:383] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.071420 6990 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:03.071843 7609 raft_consensus.cc:738] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.072525 7609 consensus_queue.cc:260] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.073737 7602 tablet_bootstrap.cc:654] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:57:55.674340 7667 debug-util.cc:398] Leaking SignalData structure 0x7b080001ca00 after lost signal to thread 7607
W20250110 20:58:03.081478 7667 kernel_stack_watchdog.cc:198] Thread 7607 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:636 for 279ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250110 20:57:54.460024 7669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:58:03.085132 7609 ts_tablet_manager.cc:1428] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20250110 20:58:03.089112 7617 tablet_replica.cc:331] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:03.089205 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet e73e60dfdab34bee95fd3bfc2fede7be (table default.table_0433916f941242f994c050f80847e23e [id=fc80a9112e4c4aa49f819343c2250505]) successfully deleted
I20250110 20:58:03.089818 7617 raft_consensus.cc:2238] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.090245 7617 raft_consensus.cc:2267] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.091820 7617 ts_tablet_manager.cc:1905] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.096305 7609 tablet_bootstrap.cc:492] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:03.097759 7615 ts_tablet_manager.cc:1918] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20250110 20:58:03.100273 7607 thread.cc:635] GCE (cloud detector) Time spent creating pthread: real 8.707s user 0.710s sys 1.254s
W20250110 20:58:03.100627 7607 thread.cc:601] GCE (cloud detector) Time spent starting thread: real 8.708s user 0.710s sys 1.254s
I20250110 20:58:03.100970 7615 log.cc:1198] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/ac84d3932d5e421fa0b641419eeebe6a
I20250110 20:58:03.106628 7615 ts_tablet_manager.cc:1939] T ac84d3932d5e421fa0b641419eeebe6a P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:03.107772 7616 ts_tablet_manager.cc:1918] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:03.112674 7602 tablet_bootstrap.cc:492] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:03.113108 7602 ts_tablet_manager.cc:1397] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.095s user 0.006s sys 0.004s
I20250110 20:58:03.115180 7602 raft_consensus.cc:357] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.096561 7606 tablet_bootstrap.cc:492] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.110073 7749 raft_consensus.cc:491] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:03.116020 7616 log.cc:1198] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/77661375951e456899d2f9052dc680e7
I20250110 20:58:03.116075 7749 raft_consensus.cc:513] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.118132 7749 leader_election.cc:290] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:03.120385 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:03.121533 7607 server_base.cc:1034] running on GCE node
I20250110 20:58:03.121680 7616 ts_tablet_manager.cc:1939] T 77661375951e456899d2f9052dc680e7 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
W20250110 20:58:03.121698 7436 leader_election.cc:343] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:03.122859 7606 tablet_bootstrap.cc:654] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.118633 7954 raft_consensus.cc:491] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:03.123281 7954 raft_consensus.cc:513] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.124975 7954 leader_election.cc:290] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.119217 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:03.126137 7210 raft_consensus.cc:2463] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:03.126747 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:03.127290 7500 raft_consensus.cc:2463] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:03.127305 7435 leader_election.cc:304] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:03.129341 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:03.127453 7602 raft_consensus.cc:383] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.129079 7145 leader_election.cc:304] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
W20250110 20:58:03.129676 7974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250110 20:58:03.130263 7954 raft_consensus.cc:2798] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:03.130564 7954 raft_consensus.cc:491] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:03.130843 7954 raft_consensus.cc:3054] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.131829 7607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250110 20:58:03.133721 7145 leader_election.cc:343] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:03.138005 7954 raft_consensus.cc:513] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.138695 7602 raft_consensus.cc:738] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.141069 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:03.128085 7615 tablet_replica.cc:331] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:03.128691 7609 tablet_bootstrap.cc:654] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:03.142395 7145 leader_election.cc:343] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:03.128767 7749 raft_consensus.cc:2798] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:03.142977 7749 raft_consensus.cc:491] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:03.143337 7749 raft_consensus.cc:3054] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.143713 7615 raft_consensus.cc:2238] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.146885 7606 tablet_bootstrap.cc:492] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.147264 7606 ts_tablet_manager.cc:1397] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.001s
I20250110 20:58:03.140864 7602 consensus_queue.cc:260] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.147749 7615 raft_consensus.cc:2267] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.152789 7616 tablet_replica.cc:331] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:03.152804 7607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250110 20:58:03.153626 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:03.155071 7616 raft_consensus.cc:2238] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.151654 7602 ts_tablet_manager.cc:1428] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.038s user 0.004s sys 0.000s
I20250110 20:58:03.152246 7954 leader_election.cc:290] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.156042 7616 raft_consensus.cc:2267] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.159572 7606 raft_consensus.cc:357] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.157661 7749 raft_consensus.cc:513] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.160817 7500 raft_consensus.cc:2388] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 244d9ba396f549f2a5015792b6c5fff8 in current term 1: Already voted for candidate c3b63d95ca734d8a9803b96bfd94f307 in this term.
I20250110 20:58:03.162298 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:03.162981 7210 raft_consensus.cc:2388] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c3b63d95ca734d8a9803b96bfd94f307 in current term 1: Already voted for candidate 244d9ba396f549f2a5015792b6c5fff8 in this term.
I20250110 20:58:03.164194 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:58:03.164999 7145 leader_election.cc:304] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:03.165838 7954 raft_consensus.cc:2743] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 20:58:03.166198 7749 leader_election.cc:290] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:03.166546 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 77661375951e456899d2f9052dc680e7 (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:03.167503 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:03.167991 7616 ts_tablet_manager.cc:1905] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:58:03.168514 7436 leader_election.cc:343] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:03.168581 7607 hybrid_clock.cc:648] HybridClock initialized: now 1736542683168480 us; error 94 us; skew 500 ppm
I20250110 20:58:03.168861 7436 leader_election.cc:304] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:03.169703 7607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250110 20:58:03.169791 7749 raft_consensus.cc:2743] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 20:58:03.160236 7606 raft_consensus.cc:383] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.172835 7606 raft_consensus.cc:738] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.174407 7609 tablet_bootstrap.cc:492] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:03.174849 7609 ts_tablet_manager.cc:1397] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.000s
I20250110 20:58:03.176926 7609 raft_consensus.cc:357] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.180347 7606 consensus_queue.cc:260] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.184064 6993 master_service.cc:946] ReplaceTablet: received request to replace tablet 43834def62374039abcbcab349edfe2e from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:03.190997 7606 ts_tablet_manager.cc:1428] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.043s user 0.004s sys 0.002s
I20250110 20:58:03.193300 7609 raft_consensus.cc:383] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.194027 7609 raft_consensus.cc:738] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.196278 7609 consensus_queue.cc:260] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.206099 7914 raft_consensus.cc:491] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:03.206507 7914 raft_consensus.cc:513] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.208220 7914 leader_election.cc:290] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:03.201087 7606 tablet_bootstrap.cc:492] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.209605 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8d5bf6a9694240b10e1b1c178b528b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:03.210189 7210 raft_consensus.cc:2463] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:03.212355 7615 ts_tablet_manager.cc:1905] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.213291 7435 leader_election.cc:304] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:03.215876 7190 tablet_service.cc:1467] Processing CreateTablet for tablet 47accefc003b4e2eae08990302350167 (DEFAULT_TABLE table=default.table_e760ae0763c04467a8fea38edff37362 [id=150bdbba49cc4d7b80896013ce4acadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.216542 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8d5bf6a9694240b10e1b1c178b528b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:03.216815 7190 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47accefc003b4e2eae08990302350167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.217152 7353 raft_consensus.cc:2463] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:03.222712 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (DEFAULT_TABLE table=default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.223634 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12429ac2edaf49b9bdddb611e97ef1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.227458 6985 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_054540086d2e402da844a531cfeb7011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:03.228478 7331 tablet_service.cc:1467] Processing CreateTablet for tablet 47accefc003b4e2eae08990302350167 (DEFAULT_TABLE table=default.table_e760ae0763c04467a8fea38edff37362 [id=150bdbba49cc4d7b80896013ce4acadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.229373 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47accefc003b4e2eae08990302350167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.229200 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 47accefc003b4e2eae08990302350167 (DEFAULT_TABLE table=default.table_e760ae0763c04467a8fea38edff37362 [id=150bdbba49cc4d7b80896013ce4acadd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.230103 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47accefc003b4e2eae08990302350167. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.233580 7914 raft_consensus.cc:2798] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:03.233906 7914 raft_consensus.cc:491] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:03.234205 7914 raft_consensus.cc:3054] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.178995 7617 ts_tablet_manager.cc:1918] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:03.239714 7617 log.cc:1198] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/9f62814cd3544ba6a1bbe0550dd8734c
I20250110 20:58:03.240378 7617 ts_tablet_manager.cc:1939] T 9f62814cd3544ba6a1bbe0550dd8734c P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:03.242408 7500 raft_consensus.cc:1270] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Refusing update from remote peer 244d9ba396f549f2a5015792b6c5fff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:03.243629 7868 consensus_queue.cc:1035] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:03.245122 7607 webserver.cc:458] Webserver started at http://127.4.171.253:33297/ using document root <none> and password file <none>
I20250110 20:58:03.247120 7607 fs_manager.cc:362] Metadata directory not provided
I20250110 20:58:03.247459 7607 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250110 20:58:03.242926 7353 raft_consensus.cc:1270] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Refusing update from remote peer 244d9ba396f549f2a5015792b6c5fff8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:03.260326 7950 consensus_queue.cc:1035] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:03.261579 7868 consensus_queue.cc:1035] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:03.261873 6993 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 43834def62374039abcbcab349edfe2e
I20250110 20:58:03.263339 6993 catalog_manager.cc:6596] ReplaceTablet: tablet 43834def62374039abcbcab349edfe2e deleted and replaced by tablet 533494271bfa4d7486026a00767cf879
I20250110 20:58:03.264756 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 43834def62374039abcbcab349edfe2e with delete_type TABLET_DATA_DELETED (replaced by tablet 533494271bfa4d7486026a00767cf879) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:03.264850 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 43834def62374039abcbcab349edfe2e with delete_type TABLET_DATA_DELETED (replaced by tablet 533494271bfa4d7486026a00767cf879) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:03.269090 7602 tablet_bootstrap.cc:492] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:03.277603 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 43834def62374039abcbcab349edfe2e with delete_type TABLET_DATA_DELETED (replaced by tablet 533494271bfa4d7486026a00767cf879) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:03.278813 6993 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:03.256991 7606 tablet_bootstrap.cc:654] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.281611 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (DEFAULT_TABLE table=default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.282666 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12429ac2edaf49b9bdddb611e97ef1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.286775 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (DEFAULT_TABLE table=default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.287742 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12429ac2edaf49b9bdddb611e97ef1c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.298936 7950 consensus_queue.cc:1035] T 721c67636cbc4fe98742a00b970f3701 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:03.314656 7189 tablet_service.cc:1467] Processing CreateTablet for tablet a580b2f85d854ff9979333b409bd6a77 (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.320477 7606 tablet_bootstrap.cc:492] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.320958 7606 ts_tablet_manager.cc:1397] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.120s user 0.007s sys 0.003s
I20250110 20:58:03.322098 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 9f62814cd3544ba6a1bbe0550dd8734c (table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]) successfully deleted
I20250110 20:58:03.322993 7617 tablet_replica.cc:331] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:03.323228 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a580b2f85d854ff9979333b409bd6a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.323652 7617 raft_consensus.cc:2238] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.325464 7333 tablet_service.cc:1467] Processing CreateTablet for tablet a580b2f85d854ff9979333b409bd6a77 (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.325805 7617 raft_consensus.cc:2267] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.326370 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a580b2f85d854ff9979333b409bd6a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.330257 7478 tablet_service.cc:1467] Processing CreateTablet for tablet a580b2f85d854ff9979333b409bd6a77 (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.328814 7914 raft_consensus.cc:513] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.331315 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a580b2f85d854ff9979333b409bd6a77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.331776 7353 raft_consensus.cc:3054] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.334493 7914 leader_election.cc:290] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:03.334957 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8d5bf6a9694240b10e1b1c178b528b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:03.335500 7351 raft_consensus.cc:3054] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.333093 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8d5bf6a9694240b10e1b1c178b528b" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:03.338757 7210 raft_consensus.cc:3054] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.342278 7351 raft_consensus.cc:2463] T fb8d5bf6a9694240b10e1b1c178b528b P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:03.344403 7210 raft_consensus.cc:2463] T fb8d5bf6a9694240b10e1b1c178b528b P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:03.345485 7435 leader_election.cc:304] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:03.347857 7749 raft_consensus.cc:2798] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:03.348444 7911 consensus_queue.cc:1035] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:03.348734 7617 ts_tablet_manager.cc:1905] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.349937 7749 raft_consensus.cc:695] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:58:03.350795 7749 consensus_queue.cc:237] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.361569 6993 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:03.370182 7602 tablet_bootstrap.cc:654] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.324146 7606 raft_consensus.cc:357] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.371443 7606 raft_consensus.cc:383] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.371778 7606 raft_consensus.cc:738] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.372593 7606 consensus_queue.cc:260] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.373725 7609 ts_tablet_manager.cc:1428] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.199s user 0.005s sys 0.000s
I20250110 20:58:03.374413 7990 tablet.cc:1721] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:03.381208 6993 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:03.390930 7991 tablet.cc:1721] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:03.395432 6989 catalog_manager.cc:2149] created HMS catalog entry for table [id=c21c62e71511487db217a6f65b6abb75]
I20250110 20:58:03.394441 7607 fs_manager.cc:714] Time spent opening directory manager: real 0.105s user 0.005s sys 0.001s
I20250110 20:58:03.408826 7615 ts_tablet_manager.cc:1918] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:03.392812 7606 ts_tablet_manager.cc:1428] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.072s user 0.005s sys 0.000s
I20250110 20:58:03.409785 7609 tablet_bootstrap.cc:492] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:03.410686 7606 tablet_bootstrap.cc:492] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.416041 7606 tablet_bootstrap.cc:654] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.389086 7914 consensus_queue.cc:1035] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:03.420517 7615 log.cc:1198] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/f25437547b454eafbcf56e2b6c3b348a
I20250110 20:58:03.422513 6989 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 80) took 2672 ms. Trace:
I20250110 20:58:03.425710 7615 ts_tablet_manager.cc:1939] T f25437547b454eafbcf56e2b6c3b348a P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:03.422859 6989 rpcz_store.cc:276] 0110 20:58:00.749787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:00.753964 (+ 4177us) service_pool.cc:225] Handling call
0110 20:58:02.566622 (+1812658us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:58:02.568505 (+ 1883us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:58:03.395270 (+826765us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:03.421694 (+ 26424us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:03.422057 (+ 363us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:03.422434 (+ 377us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:58:03.397181 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:58:03.397548 (+ 367us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:58:03.397567 (+ 19us) write_op.cc:435] Acquired schema lock
0110 20:58:03.397578 (+ 11us) tablet.cc:659] Decoding operations
0110 20:58:03.397834 (+ 256us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:58:03.397865 (+ 31us) write_op.cc:641] Partition lock acquired for write op
0110 20:58:03.397885 (+ 20us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:58:03.398063 (+ 178us) tablet.cc:698] Row locks acquired
0110 20:58:03.398076 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:58:03.398168 (+ 92us) write_op.cc:270] Start()
0110 20:58:03.398239 (+ 71us) write_op.cc:276] Timestamp: P: 1736542683398062 usec, L: 0
0110 20:58:03.398253 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:58:03.400384 (+ 2131us) log.cc:844] Serialized 433 byte log entry
0110 20:58:03.419652 (+ 19268us) op_driver.cc:464] REPLICATION: finished
0110 20:58:03.420172 (+ 520us) write_op.cc:301] APPLY: starting
0110 20:58:03.420226 (+ 54us) tablet.cc:1364] starting BulkCheckPresence
0110 20:58:03.420321 (+ 95us) tablet.cc:1367] finished BulkCheckPresence
0110 20:58:03.420414 (+ 93us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:58:03.420764 (+ 350us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:58:03.420894 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:58:03.420940 (+ 46us) write_op.cc:312] APPLY: finished
0110 20:58:03.421162 (+ 222us) log.cc:844] Serialized 29 byte log entry
0110 20:58:03.421290 (+ 128us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:03.421353 (+ 63us) write_op.cc:454] Released schema lock
0110 20:58:03.421382 (+ 29us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":405901,"Hive Metastore.run_cpu_time_us":792,"Hive Metastore.run_wall_time_us":419977,"child_traces":[["op",{"apply.queue_time_us":363,"num_ops":2,"prepare.queue_time_us":240,"prepare.run_cpu_time_us":1809,"prepare.run_wall_time_us":3580,"raft.queue_time_us":71,"raft.run_cpu_time_us":12,"raft.run_wall_time_us":12,"replication_time_us":21319}]]}
I20250110 20:58:03.433597 7997 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250110 20:58:03.433476 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:03.443980 7607 fs_manager.cc:730] Time spent opening block manager: real 0.042s user 0.002s sys 0.001s
I20250110 20:58:03.445998 6993 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:03.444415 7607 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data,/tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
uuid: "f9bb61bdf30044a1b482d6a66e29435f"
format_stamp: "Formatted at 2025-01-10 20:57:42 on dist-test-slave-98kh"
I20250110 20:58:03.448072 7609 tablet_bootstrap.cc:654] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.447677 7607 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250110 20:58:03.456964 7602 tablet_bootstrap.cc:492] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:03.457990 7602 ts_tablet_manager.cc:1397] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.189s user 0.008s sys 0.001s
I20250110 20:58:03.465795 7615 tablet_replica.cc:331] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:03.466512 7615 raft_consensus.cc:2238] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.473834 7616 ts_tablet_manager.cc:1918] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:03.474068 7615 raft_consensus.cc:2267] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.474218 7616 log.cc:1198] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/ac84d3932d5e421fa0b641419eeebe6a
I20250110 20:58:03.474872 7616 ts_tablet_manager.cc:1939] T ac84d3932d5e421fa0b641419eeebe6a P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:03.477813 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:58:03.481985 7609 tablet_bootstrap.cc:492] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:03.482378 7609 ts_tablet_manager.cc:1397] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.000s
I20250110 20:58:03.483865 7606 tablet_bootstrap.cc:492] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.484239 7606 ts_tablet_manager.cc:1397] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.003s
I20250110 20:58:03.484584 7609 raft_consensus.cc:357] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.485191 7609 raft_consensus.cc:383] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.485981 7615 ts_tablet_manager.cc:1905] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.486732 7606 raft_consensus.cc:357] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.494474 6989 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:03.495571 7617 ts_tablet_manager.cc:1918] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:03.495924 7617 log.cc:1198] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/164f26db984343a390f253aba1219f4a
I20250110 20:58:03.495187 6989 catalog_manager.cc:5526] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:58:03.496557 7617 ts_tablet_manager.cc:1939] T 164f26db984343a390f253aba1219f4a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:03.487349 7606 raft_consensus.cc:383] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.497224 7606 raft_consensus.cc:738] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
W20250110 20:58:03.501494 7287 consensus_peers.cc:487] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Couldn't send request to peer 244d9ba396f549f2a5015792b6c5fff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f25437547b454eafbcf56e2b6c3b348a. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:58:03.497830 7606 consensus_queue.cc:260] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.511018 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet a580b2f85d854ff9979333b409bd6a77 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:03.514134 7606 ts_tablet_manager.cc:1428] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.030s user 0.006s sys 0.000s
I20250110 20:58:03.517637 7606 tablet_bootstrap.cc:492] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.531219 7616 tablet_replica.cc:331] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:03.531736 7617 tablet_replica.cc:331] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:03.531960 7616 raft_consensus.cc:2238] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.532446 7616 raft_consensus.cc:2267] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.532438 7617 raft_consensus.cc:2238] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.533532 7609 raft_consensus.cc:738] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.534293 7609 consensus_queue.cc:260] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.540885 7617 raft_consensus.cc:2267] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.541865 7602 raft_consensus.cc:357] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.543881 7617 ts_tablet_manager.cc:1905] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.547537 7602 raft_consensus.cc:383] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.548446 7602 raft_consensus.cc:738] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.549933 7616 ts_tablet_manager.cc:1905] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.551007 7602 consensus_queue.cc:260] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.555931 7606 tablet_bootstrap.cc:654] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.561738 7609 ts_tablet_manager.cc:1428] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.079s user 0.006s sys 0.000s
I20250110 20:58:03.562469 7609 tablet_bootstrap.cc:492] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:03.553692 8002 tablet.cc:1721] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:03.568773 7602 ts_tablet_manager.cc:1428] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.108s user 0.005s sys 0.000s
I20250110 20:58:03.573453 7602 tablet_bootstrap.cc:492] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:03.578382 6986 catalog_manager.cc:2149] created HMS catalog entry for table [id=f974e7b92b494a63a9544aaad6c79f07]
I20250110 20:58:03.580276 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 164f26db984343a390f253aba1219f4a (table default.table_7a871347a918413297f7da917134a92e [id=2b85c6667e194f818c41937e2307c37f]) successfully deleted
I20250110 20:58:03.589587 7606 tablet_bootstrap.cc:492] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.589980 7606 ts_tablet_manager.cc:1397] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.073s user 0.009s sys 0.000s
I20250110 20:58:03.590103 7609 tablet_bootstrap.cc:654] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.591919 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:03.592142 7606 raft_consensus.cc:357] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.592754 7602 tablet_bootstrap.cc:654] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.592775 7606 raft_consensus.cc:383] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.593014 7606 raft_consensus.cc:738] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.593178 6990 catalog_manager.cc:6596] ReplaceTablet: tablet a580b2f85d854ff9979333b409bd6a77 deleted and replaced by tablet 1c82412f7ba946b9b0dcffd8fa883353
I20250110 20:58:03.595633 7926 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:03.595829 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet a580b2f85d854ff9979333b409bd6a77 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82412f7ba946b9b0dcffd8fa883353) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:03.596027 7926 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:03.597653 6989 master_service.cc:946] ReplaceTablet: received request to replace tablet 72de597ddab241fbbb82fc04e40385c5 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:03.597927 7926 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:03.599341 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:58:03.600100 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:03.600493 7287 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:03.601507 7926 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:03.605695 7606 consensus_queue.cc:260] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.607883 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet a580b2f85d854ff9979333b409bd6a77 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82412f7ba946b9b0dcffd8fa883353) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:03.608808 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet a580b2f85d854ff9979333b409bd6a77 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82412f7ba946b9b0dcffd8fa883353) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:03.615096 7606 ts_tablet_manager.cc:1428] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20250110 20:58:03.628386 7602 tablet_bootstrap.cc:492] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:03.628790 7602 ts_tablet_manager.cc:1397] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.007s
I20250110 20:58:03.628870 7609 tablet_bootstrap.cc:492] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:03.629238 7609 ts_tablet_manager.cc:1397] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.000s
I20250110 20:58:03.631377 7609 raft_consensus.cc:357] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.632045 7609 raft_consensus.cc:383] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.632094 7606 tablet_bootstrap.cc:492] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.632299 7609 raft_consensus.cc:738] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.633538 7602 raft_consensus.cc:357] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.636219 7602 raft_consensus.cc:383] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.636585 7602 raft_consensus.cc:738] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.637213 7602 consensus_queue.cc:260] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.640587 7602 ts_tablet_manager.cc:1428] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.012s user 0.004s sys 0.001s
I20250110 20:58:03.637482 7606 tablet_bootstrap.cc:654] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.646328 7616 ts_tablet_manager.cc:1918] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:03.646692 7616 log.cc:1198] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/f25437547b454eafbcf56e2b6c3b348a
I20250110 20:58:03.647354 7616 ts_tablet_manager.cc:1939] T f25437547b454eafbcf56e2b6c3b348a P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:03.648018 7602 tablet_bootstrap.cc:492] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:03.650902 7609 consensus_queue.cc:260] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.657034 7602 tablet_bootstrap.cc:654] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.659050 7609 ts_tablet_manager.cc:1428] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.030s user 0.004s sys 0.000s
I20250110 20:58:03.659834 7609 tablet_bootstrap.cc:492] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:03.674113 7616 tablet_replica.cc:331] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
W20250110 20:58:03.679566 6989 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:03.681242 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet f25437547b454eafbcf56e2b6c3b348a (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:03.684402 7602 tablet_bootstrap.cc:492] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:03.684782 7602 ts_tablet_manager.cc:1397] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.037s user 0.009s sys 0.005s
I20250110 20:58:03.685420 7609 tablet_bootstrap.cc:654] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.686830 7602 raft_consensus.cc:357] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.687644 7602 raft_consensus.cc:383] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.674791 7616 raft_consensus.cc:2238] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.687881 7602 raft_consensus.cc:738] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.688253 7616 raft_consensus.cc:2267] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.688450 7602 consensus_queue.cc:260] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.691201 6986 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 83) took 2873 ms. Trace:
I20250110 20:58:03.691694 7616 ts_tablet_manager.cc:1905] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.691519 6986 rpcz_store.cc:276] 0110 20:58:00.817573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:00.817697 (+ 124us) service_pool.cc:225] Handling call
0110 20:58:02.574813 (+1757116us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:58:02.576460 (+ 1647us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:58:03.578235 (+1001775us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:03.690493 (+112258us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:03.690811 (+ 318us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:03.691146 (+ 335us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:58:03.580477 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:58:03.580824 (+ 347us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:58:03.580845 (+ 21us) write_op.cc:435] Acquired schema lock
0110 20:58:03.580858 (+ 13us) tablet.cc:659] Decoding operations
0110 20:58:03.581132 (+ 274us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:58:03.581165 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0110 20:58:03.581187 (+ 22us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:58:03.581353 (+ 166us) tablet.cc:698] Row locks acquired
0110 20:58:03.581366 (+ 13us) write_op.cc:260] PREPARE: finished
0110 20:58:03.581485 (+ 119us) write_op.cc:270] Start()
0110 20:58:03.581561 (+ 76us) write_op.cc:276] Timestamp: P: 1736542683581358 usec, L: 0
0110 20:58:03.581575 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:58:03.582001 (+ 426us) log.cc:844] Serialized 433 byte log entry
0110 20:58:03.687116 (+105115us) op_driver.cc:464] REPLICATION: finished
0110 20:58:03.687513 (+ 397us) write_op.cc:301] APPLY: starting
0110 20:58:03.687562 (+ 49us) tablet.cc:1364] starting BulkCheckPresence
0110 20:58:03.687656 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0110 20:58:03.687801 (+ 145us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:58:03.688917 (+ 1116us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:58:03.689065 (+ 148us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:58:03.689103 (+ 38us) write_op.cc:312] APPLY: finished
0110 20:58:03.689287 (+ 184us) log.cc:844] Serialized 29 byte log entry
0110 20:58:03.690212 (+ 925us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:03.690274 (+ 62us) write_op.cc:454] Released schema lock
0110 20:58:03.690304 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"Hive Metastore.queue_time_us":818009,"Hive Metastore.run_cpu_time_us":765,"Hive Metastore.run_wall_time_us":183093,"child_traces":[["op",{"apply.queue_time_us":256,"num_ops":2,"prepare.queue_time_us":541,"prepare.run_cpu_time_us":1704,"prepare.run_wall_time_us":1704,"replication_time_us":105460}]]}
I20250110 20:58:03.695305 7602 ts_tablet_manager.cc:1428] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20250110 20:58:03.696738 6986 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_3e0bbc862f1a4a84816adb5841844953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:03.568773 7615 ts_tablet_manager.cc:1918] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:03.698547 7615 log.cc:1198] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/43834def62374039abcbcab349edfe2e
I20250110 20:58:03.699962 7615 ts_tablet_manager.cc:1939] T 43834def62374039abcbcab349edfe2e P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:03.702358 7615 tablet_replica.cc:331] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:03.702364 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 43834def62374039abcbcab349edfe2e (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:03.704821 7606 tablet_bootstrap.cc:492] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.705196 7606 ts_tablet_manager.cc:1397] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.001s
I20250110 20:58:03.707267 7606 raft_consensus.cc:357] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.707886 7606 raft_consensus.cc:383] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.708129 7606 raft_consensus.cc:738] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.708693 7606 consensus_queue.cc:260] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.713614 7609 tablet_bootstrap.cc:492] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:03.714041 7609 ts_tablet_manager.cc:1397] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.001s
I20250110 20:58:03.715761 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 533494271bfa4d7486026a00767cf879 (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.716084 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 533494271bfa4d7486026a00767cf879 (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.716655 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533494271bfa4d7486026a00767cf879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.718250 7609 raft_consensus.cc:357] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.718873 7609 raft_consensus.cc:383] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.719122 7609 raft_consensus.cc:738] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.719664 7609 consensus_queue.cc:260] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.723171 7606 ts_tablet_manager.cc:1428] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.018s user 0.004s sys 0.000s
I20250110 20:58:03.725610 7609 ts_tablet_manager.cc:1428] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20250110 20:58:03.726469 7606 tablet_bootstrap.cc:492] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.730283 6989 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
W20250110 20:58:03.737637 6989 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 12429ac2edaf49b9bdddb611e97ef1c3 consensus state has no leader
I20250110 20:58:03.738657 7606 tablet_bootstrap.cc:654] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.744654 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533494271bfa4d7486026a00767cf879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.751803 7616 ts_tablet_manager.cc:1918] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:03.752174 7616 log.cc:1198] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/43834def62374039abcbcab349edfe2e
I20250110 20:58:03.752872 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 18a3bdfc63a046f08ff4885e335c9758 (DEFAULT_TABLE table=default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.753978 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a3bdfc63a046f08ff4885e335c9758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.761482 7616 ts_tablet_manager.cc:1939] T 43834def62374039abcbcab349edfe2e P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:03.765929 7616 tablet_replica.cc:331] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:03.752099 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 18a3bdfc63a046f08ff4885e335c9758 (DEFAULT_TABLE table=default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.766577 7616 raft_consensus.cc:2238] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.766957 7616 raft_consensus.cc:2267] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.767885 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a3bdfc63a046f08ff4885e335c9758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.769563 7616 ts_tablet_manager.cc:1905] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.771687 7609 tablet_bootstrap.cc:492] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:03.772469 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 43834def62374039abcbcab349edfe2e (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:03.718818 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 533494271bfa4d7486026a00767cf879 (DEFAULT_TABLE table=default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.774086 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 533494271bfa4d7486026a00767cf879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.776499 7609 tablet_bootstrap.cc:654] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.719784 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 18a3bdfc63a046f08ff4885e335c9758 (DEFAULT_TABLE table=default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.778028 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18a3bdfc63a046f08ff4885e335c9758. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.769656 7606 tablet_bootstrap.cc:492] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.782157 7606 ts_tablet_manager.cc:1397] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.056s user 0.010s sys 0.000s
I20250110 20:58:03.784308 7606 raft_consensus.cc:357] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.784927 7606 raft_consensus.cc:383] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.785174 7606 raft_consensus.cc:738] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.789182 7616 ts_tablet_manager.cc:1918] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:03.790041 7926 raft_consensus.cc:491] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:03.797089 7926 raft_consensus.cc:513] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.798966 7926 leader_election.cc:290] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.800459 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12429ac2edaf49b9bdddb611e97ef1c3" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:03.801064 7210 raft_consensus.cc:2463] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:03.802466 7287 leader_election.cc:304] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:03.803203 7926 raft_consensus.cc:2798] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:03.801146 7609 tablet_bootstrap.cc:492] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:03.803503 7926 raft_consensus.cc:491] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:03.803546 7616 log.cc:1198] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:03.803670 7609 ts_tablet_manager.cc:1397] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.010s
I20250110 20:58:03.803786 7926 raft_consensus.cc:3054] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.806239 7609 raft_consensus.cc:357] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.806892 7609 raft_consensus.cc:383] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.807129 7609 raft_consensus.cc:738] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.807694 7609 consensus_queue.cc:260] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.804136 7616 ts_tablet_manager.cc:1939] T a580b2f85d854ff9979333b409bd6a77 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:03.809669 7926 raft_consensus.cc:513] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.812170 7609 ts_tablet_manager.cc:1428] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250110 20:58:03.812227 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12429ac2edaf49b9bdddb611e97ef1c3" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:03.812722 7210 raft_consensus.cc:3054] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.813544 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12429ac2edaf49b9bdddb611e97ef1c3" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:03.814152 7500 raft_consensus.cc:2463] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:03.825268 7210 raft_consensus.cc:2463] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:03.828305 7287 leader_election.cc:304] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
W20250110 20:58:03.839921 6980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.254:36415, user_credentials={real_user=slave}} blocked reactor thread for 66472.3us
I20250110 20:58:03.840382 6993 master_service.cc:946] ReplaceTablet: received request to replace tablet 3d4511e84bd7406cafe8691e0216ad93 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:03.843328 6991 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33774 (request call id 65) took 4732 ms. Trace:
I20250110 20:58:03.843482 7926 leader_election.cc:290] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.843950 7926 raft_consensus.cc:491] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250110 20:58:03.844269 6987 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_eed9317ae3914199a2cdc0d7e112371f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:58:03.844332 7926 raft_consensus.cc:513] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.844892 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12429ac2edaf49b9bdddb611e97ef1c3" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:03.845475 7500 raft_consensus.cc:3054] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.843637 6991 rpcz_store.cc:276] 0110 20:57:59.110916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:59.111061 (+ 145us) service_pool.cc:225] Handling call
0110 20:57:59.159152 (+ 48091us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x7b5400003390. stack: 00007f728d56d853 00007f728d56d750 00007f728d0e2584 00000000004fcd24 00000000004fccb9 00007f728d3e32d1 00007f728d562d60 00007f7296173c26 00007f7296173673 00007f729628f64f 00007f7293ff7b44 00007f7293ff7ad2 00007f7293ff7a01 00007f7293ff797c 00007f7293ff6c32 00007f72922b5dfc
0110 20:58:00.622356 (+1463204us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:58:03.843128 (+3220772us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":894117,"Hive Metastore.run_cpu_time_us":334,"Hive Metastore.run_wall_time_us":188727,"mutex_wait_us":10078,"spinlock_wait_cycles":8162560}
I20250110 20:58:03.848904 6994 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:33774 (request call id 74) took 3955 ms. Trace:
I20250110 20:58:03.849213 6994 rpcz_store.cc:276] 0110 20:57:59.893281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:57:59.893399 (+ 118us) service_pool.cc:225] Handling call
0110 20:58:01.684830 (+1791431us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:58:03.848741 (+2163911us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":30671,"Hive Metastore.run_cpu_time_us":291,"Hive Metastore.run_wall_time_us":629599}
I20250110 20:58:03.850409 7954 raft_consensus.cc:491] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:03.850848 7954 raft_consensus.cc:513] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.852632 7954 leader_election.cc:290] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.854442 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47accefc003b4e2eae08990302350167" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:03.854976 7501 raft_consensus.cc:2463] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
W20250110 20:58:03.855736 6985 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_054540086d2e402da844a531cfeb7011 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:58:03.855983 8015 raft_consensus.cc:2798] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:03.855948 7145 leader_election.cc:304] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:03.856418 8015 raft_consensus.cc:695] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:03.856618 7954 raft_consensus.cc:2798] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:03.856899 7954 raft_consensus.cc:491] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:03.857174 7954 raft_consensus.cc:3054] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.858090 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47accefc003b4e2eae08990302350167" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:03.858677 7353 raft_consensus.cc:2463] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
W20250110 20:58:03.858997 6986 catalog_manager.cc:6885] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_3e0bbc862f1a4a84816adb5841844953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250110 20:58:03.861796 7606 consensus_queue.cc:260] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.863554 7500 raft_consensus.cc:2463] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:03.866667 7954 raft_consensus.cc:513] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.868611 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:03.869290 7500 raft_consensus.cc:2388] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ef4ce3d44cc486fb0ee0ad85127011b in current term 1: Already voted for candidate c3b63d95ca734d8a9803b96bfd94f307 in this term.
I20250110 20:58:03.869661 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47accefc003b4e2eae08990302350167" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:03.870177 7353 raft_consensus.cc:3054] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.870985 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47accefc003b4e2eae08990302350167" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:03.871397 7615 raft_consensus.cc:2238] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:03.871465 7500 raft_consensus.cc:3054] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.871953 7615 raft_consensus.cc:2267] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:03.873641 7954 leader_election.cc:290] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.876963 7353 raft_consensus.cc:2463] T 47accefc003b4e2eae08990302350167 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:03.877805 7145 leader_election.cc:304] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:03.857138 8015 consensus_queue.cc:237] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.881572 7954 raft_consensus.cc:2798] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
W20250110 20:58:03.882056 7287 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.193:36745, user_credentials={real_user=slave}} blocked reactor thread for 53859.9us
I20250110 20:58:03.884769 7954 raft_consensus.cc:695] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:58:03.884927 6994 master_service.cc:946] ReplaceTablet: received request to replace tablet f3e994b6ed3e4692a4ba3cfcf3b72648 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:03.873351 7926 leader_election.cc:290] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:03.885531 7954 consensus_queue.cc:237] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.889731 7615 ts_tablet_manager.cc:1905] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:03.890143 7500 raft_consensus.cc:2463] T 47accefc003b4e2eae08990302350167 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:03.892095 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833cc8d477374f51a67422cbe051254e" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:03.892830 7210 raft_consensus.cc:2388] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9ef4ce3d44cc486fb0ee0ad85127011b in current term 1: Already voted for candidate 244d9ba396f549f2a5015792b6c5fff8 in this term.
I20250110 20:58:03.899791 6991 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_7df54f46d73f40ef8391382214ed929a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:03.903666 7606 ts_tablet_manager.cc:1428] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.121s user 0.005s sys 0.003s
I20250110 20:58:03.905642 7606 tablet_bootstrap.cc:492] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.911053 7606 tablet_bootstrap.cc:654] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.919418 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet a580b2f85d854ff9979333b409bd6a77 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:58:03.923959 7606 tablet_bootstrap.cc:492] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.924320 7606 ts_tablet_manager.cc:1397] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20250110 20:58:03.926411 7606 raft_consensus.cc:357] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.927029 7606 raft_consensus.cc:383] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.927277 7606 raft_consensus.cc:738] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.927938 7606 consensus_queue.cc:260] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.930317 7606 ts_tablet_manager.cc:1428] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250110 20:58:03.931507 7606 tablet_bootstrap.cc:492] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:03.936537 7606 tablet_bootstrap.cc:654] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:03.942390 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 12429ac2edaf49b9bdddb611e97ef1c3 consensus state has no leader
I20250110 20:58:03.904712 7287 leader_election.cc:304] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:03.943662 8015 raft_consensus.cc:3054] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:03.947381 7606 tablet_bootstrap.cc:492] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:03.947739 7606 ts_tablet_manager.cc:1397] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20250110 20:58:03.947831 7602 tablet_bootstrap.cc:492] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:03.949786 7606 raft_consensus.cc:357] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.950379 7606 raft_consensus.cc:383] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.950610 7606 raft_consensus.cc:738] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.951247 7606 consensus_queue.cc:260] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.957158 7606 ts_tablet_manager.cc:1428] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250110 20:58:03.958384 6990 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:03.958923 7602 tablet_bootstrap.cc:654] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.961971 7615 ts_tablet_manager.cc:1918] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:03.957101 8015 raft_consensus.cc:2743] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:03.962378 7615 log.cc:1198] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/a580b2f85d854ff9979333b409bd6a77
W20250110 20:58:03.962810 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:03.962949 7615 ts_tablet_manager.cc:1939] T a580b2f85d854ff9979333b409bd6a77 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:03.965097 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet a580b2f85d854ff9979333b409bd6a77 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:58:03.970885 7478 tablet_service.cc:1467] Processing CreateTablet for tablet ab2f24f374cb4dfba6d9563cb6666f6c (DEFAULT_TABLE table=default.table_4dac0491728e4031866a732faeb6d42c [id=f974e7b92b494a63a9544aaad6c79f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.971563 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82412f7ba946b9b0dcffd8fa883353 (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.972039 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab2f24f374cb4dfba6d9563cb6666f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.972146 7617 ts_tablet_manager.cc:1918] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:03.972472 7617 log.cc:1198] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ec62f901ffe64faea7d3716db20c487f
I20250110 20:58:03.972501 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82412f7ba946b9b0dcffd8fa883353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.973080 7617 ts_tablet_manager.cc:1939] T ec62f901ffe64faea7d3716db20c487f P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:03.974359 7602 tablet_bootstrap.cc:492] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:03.974728 7602 ts_tablet_manager.cc:1397] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.001s
I20250110 20:58:03.974711 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82412f7ba946b9b0dcffd8fa883353 (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.975596 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82412f7ba946b9b0dcffd8fa883353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.976863 7602 raft_consensus.cc:357] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:03.977774 7479 tablet_service.cc:1514] Processing DeleteTablet for tablet a580b2f85d854ff9979333b409bd6a77 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82412f7ba946b9b0dcffd8fa883353) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:03.979084 7602 raft_consensus.cc:383] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:03.979348 7602 raft_consensus.cc:738] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:03.980077 7609 tablet_bootstrap.cc:492] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:03.984903 7609 tablet_bootstrap.cc:654] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:03.986086 7333 tablet_service.cc:1467] Processing CreateTablet for tablet ab2f24f374cb4dfba6d9563cb6666f6c (DEFAULT_TABLE table=default.table_4dac0491728e4031866a732faeb6d42c [id=f974e7b92b494a63a9544aaad6c79f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:03.987072 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab2f24f374cb4dfba6d9563cb6666f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:03.988250 7602 consensus_queue.cc:260] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:58:03.999807 6980 catalog_manager.cc:4839] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): delete failed for tablet a580b2f85d854ff9979333b409bd6a77 because the tablet was not found. No further retry: Not found: Tablet not found: a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:04.002779 7602 ts_tablet_manager.cc:1428] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250110 20:58:04.012771 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 112) took 1315 ms. Trace:
I20250110 20:58:04.011617 7188 tablet_service.cc:1467] Processing CreateTablet for tablet ab2f24f374cb4dfba6d9563cb6666f6c (DEFAULT_TABLE table=default.table_4dac0491728e4031866a732faeb6d42c [id=f974e7b92b494a63a9544aaad6c79f07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.019738 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab2f24f374cb4dfba6d9563cb6666f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.022717 7606 tablet_bootstrap.cc:492] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:04.024473 7602 tablet_bootstrap.cc:492] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:04.027714 7606 tablet_bootstrap.cc:654] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.030313 7609 tablet_bootstrap.cc:492] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:04.030684 7609 ts_tablet_manager.cc:1397] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.004s
W20250110 20:58:04.030622 7288 consensus_peers.cc:487] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Couldn't send request to peer c3b63d95ca734d8a9803b96bfd94f307. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac84d3932d5e421fa0b641419eeebe6a. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:58:04.037029 6994 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet f3e994b6ed3e4692a4ba3cfcf3b72648
I20250110 20:58:04.037477 7602 tablet_bootstrap.cc:654] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.037750 6993 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 3d4511e84bd7406cafe8691e0216ad93
I20250110 20:58:04.038350 6994 catalog_manager.cc:6596] ReplaceTablet: tablet f3e994b6ed3e4692a4ba3cfcf3b72648 deleted and replaced by tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8
I20250110 20:58:04.038920 6993 catalog_manager.cc:6596] ReplaceTablet: tablet 3d4511e84bd7406cafe8691e0216ad93 deleted and replaced by tablet f58d4b7a1ff0439a9b1a08e9e04c044d
I20250110 20:58:04.039459 7187 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4511e84bd7406cafe8691e0216ad93 with delete_type TABLET_DATA_DELETED (replaced by tablet f58d4b7a1ff0439a9b1a08e9e04c044d) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:04.040666 6989 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:04.040892 7615 tablet_replica.cc:331] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:04.041597 7615 raft_consensus.cc:2238] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:04.042378 7615 raft_consensus.cc:2267] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.044003 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e994b6ed3e4692a4ba3cfcf3b72648 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:04.044440 7615 ts_tablet_manager.cc:1905] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.013159 7617 rpcz_store.cc:276] 0110 20:58:02.697019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:02.697125 (+ 106us) service_pool.cc:225] Handling call
0110 20:58:03.531594 (+834469us) ts_tablet_manager.cc:1145] Deleting tablet ec62f901ffe64faea7d3716db20c487f
0110 20:58:03.531632 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:03.706743 (+175111us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:03.971946 (+265203us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:04.012711 (+ 40765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":833933}
I20250110 20:58:04.033336 7609 raft_consensus.cc:357] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.049786 7617 tablet_replica.cc:331] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:04.049968 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet a580b2f85d854ff9979333b409bd6a77 with delete_type TABLET_DATA_DELETED (replaced by tablet 1c82412f7ba946b9b0dcffd8fa883353) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:04.050011 7609 raft_consensus.cc:383] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.050261 7609 raft_consensus.cc:738] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.050755 7617 raft_consensus.cc:2238] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.056190 6989 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_34c642de1a56476a83c2b97f9cdd1287" } modify_external_catalogs: true
I20250110 20:58:04.056907 6989 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_34c642de1a56476a83c2b97f9cdd1287" } modify_external_catalogs: true
I20250110 20:58:04.052815 7609 consensus_queue.cc:260] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.064568 6994 catalog_manager.cc:2462] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_df681f6d6d9e483282a9b28e42817aef" } modify_external_catalogs: true
I20250110 20:58:04.065299 6994 catalog_manager.cc:2646] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33774:
table { table_name: "default.Table_df681f6d6d9e483282a9b28e42817aef" } modify_external_catalogs: true
I20250110 20:58:04.067871 8015 raft_consensus.cc:491] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.068630 7606 tablet_bootstrap.cc:492] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:04.068300 8015 raft_consensus.cc:513] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.068996 7606 ts_tablet_manager.cc:1397] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.046s user 0.009s sys 0.003s
I20250110 20:58:04.070075 8015 leader_election.cc:290] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.071730 7606 raft_consensus.cc:357] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.072384 7606 raft_consensus.cc:383] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.072613 7606 raft_consensus.cc:738] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.072878 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3bdfc63a046f08ff4885e335c9758" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:04.073467 7500 raft_consensus.cc:2463] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:04.073222 7606 consensus_queue.cc:260] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.074038 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3bdfc63a046f08ff4885e335c9758" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:04.074534 7288 leader_election.cc:304] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
W20250110 20:58:04.075277 7287 leader_election.cc:343] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:04.075662 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet ec62f901ffe64faea7d3716db20c487f (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]) successfully deleted
I20250110 20:58:04.077183 7926 raft_consensus.cc:2798] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:04.077534 7926 raft_consensus.cc:491] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:04.077569 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e994b6ed3e4692a4ba3cfcf3b72648 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:04.077795 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet f3e994b6ed3e4692a4ba3cfcf3b72648 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:04.077834 7926 raft_consensus.cc:3054] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.078173 7616 tablet_replica.cc:331] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:04.078841 7616 raft_consensus.cc:2238] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:04.079483 7609 ts_tablet_manager.cc:1428] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20250110 20:58:04.079587 7616 raft_consensus.cc:2267] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.081722 7616 ts_tablet_manager.cc:1905] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.084297 7926 raft_consensus.cc:513] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.086083 7926 leader_election.cc:290] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.087440 7209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3bdfc63a046f08ff4885e335c9758" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:04.088326 7609 tablet_bootstrap.cc:492] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
W20250110 20:58:04.088820 7287 leader_election.cc:343] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:04.094552 7606 ts_tablet_manager.cc:1428] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.025s user 0.003s sys 0.004s
I20250110 20:58:04.018082 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 1c82412f7ba946b9b0dcffd8fa883353 (DEFAULT_TABLE table=default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.099378 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c82412f7ba946b9b0dcffd8fa883353. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.104102 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4511e84bd7406cafe8691e0216ad93 with delete_type TABLET_DATA_DELETED (replaced by tablet f58d4b7a1ff0439a9b1a08e9e04c044d) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:04.108568 7606 tablet_bootstrap.cc:492] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:04.110908 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3bdfc63a046f08ff4885e335c9758" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:04.111459 7500 raft_consensus.cc:3054] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.111781 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d4511e84bd7406cafe8691e0216ad93 with delete_type TABLET_DATA_DELETED (replaced by tablet f58d4b7a1ff0439a9b1a08e9e04c044d) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:04.112134 7609 tablet_bootstrap.cc:654] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.115236 7615 ts_tablet_manager.cc:1918] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:04.115593 7615 log.cc:1198] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/3d4511e84bd7406cafe8691e0216ad93
I20250110 20:58:04.117118 7602 tablet_bootstrap.cc:492] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:04.117533 7602 ts_tablet_manager.cc:1397] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.093s user 0.009s sys 0.000s
I20250110 20:58:04.117841 7500 raft_consensus.cc:2463] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:04.118435 7606 tablet_bootstrap.cc:654] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.118806 7288 leader_election.cc:304] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:58:04.119511 7926 raft_consensus.cc:2798] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:04.119935 7926 raft_consensus.cc:695] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:04.119555 7602 raft_consensus.cc:357] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.120293 7602 raft_consensus.cc:383] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.120523 7602 raft_consensus.cc:738] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.120837 7616 ts_tablet_manager.cc:1918] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:04.120643 7926 consensus_queue.cc:237] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.121151 7616 log.cc:1198] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/f3e994b6ed3e4692a4ba3cfcf3b72648
I20250110 20:58:04.121790 7616 ts_tablet_manager.cc:1939] T f3e994b6ed3e4692a4ba3cfcf3b72648 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:04.121598 7602 consensus_queue.cc:260] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.123157 7615 ts_tablet_manager.cc:1939] T 3d4511e84bd7406cafe8691e0216ad93 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:04.126572 7602 ts_tablet_manager.cc:1428] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250110 20:58:04.051234 7617 raft_consensus.cc:2267] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.130863 7617 ts_tablet_manager.cc:1905] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.131027 7609 tablet_bootstrap.cc:492] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:04.131590 7609 ts_tablet_manager.cc:1397] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.000s
I20250110 20:58:04.131600 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:58:04.135947 7609 raft_consensus.cc:357] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.141907 7609 raft_consensus.cc:383] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.142174 7609 raft_consensus.cc:738] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.143270 7954 raft_consensus.cc:491] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.143430 7615 tablet_replica.cc:331] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:04.143692 7954 raft_consensus.cc:513] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.144131 7615 raft_consensus.cc:2238] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.144583 7615 raft_consensus.cc:2267] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.145304 7616 tablet_replica.cc:331] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:04.145534 7954 leader_election.cc:290] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.146862 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3bdfc63a046f08ff4885e335c9758" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:04.147024 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a3bdfc63a046f08ff4885e335c9758" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:04.147707 7500 raft_consensus.cc:2388] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 244d9ba396f549f2a5015792b6c5fff8 in current term 1: Already voted for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in this term.
I20250110 20:58:04.146021 7616 raft_consensus.cc:2238] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.142793 7609 consensus_queue.cc:260] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.148587 7616 raft_consensus.cc:2267] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.153895 7145 leader_election.cc:304] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8; no voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:04.154798 7954 raft_consensus.cc:3054] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.155614 7616 ts_tablet_manager.cc:1905] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.161077 7954 raft_consensus.cc:2743] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:04.162060 7609 ts_tablet_manager.cc:1428] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20250110 20:58:04.163033 7609 tablet_bootstrap.cc:492] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:04.165696 7615 ts_tablet_manager.cc:1905] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.167901 7609 tablet_bootstrap.cc:654] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.175072 7606 tablet_bootstrap.cc:492] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:04.175529 7606 ts_tablet_manager.cc:1397] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.005s
I20250110 20:58:04.176441 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet f3e994b6ed3e4692a4ba3cfcf3b72648 (table default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]) successfully deleted
I20250110 20:58:04.177837 7606 raft_consensus.cc:357] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.178552 7606 raft_consensus.cc:383] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.181495 7606 raft_consensus.cc:738] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.182224 7606 consensus_queue.cc:260] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.186823 7606 ts_tablet_manager.cc:1428] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.011s user 0.004s sys 0.000s
I20250110 20:58:04.195539 7615 ts_tablet_manager.cc:1918] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:04.195873 7615 log.cc:1198] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/f3e994b6ed3e4692a4ba3cfcf3b72648
I20250110 20:58:04.196507 7615 ts_tablet_manager.cc:1939] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:04.199904 7609 tablet_bootstrap.cc:492] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:04.200379 7609 ts_tablet_manager.cc:1397] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.005s
I20250110 20:58:04.199474 7616 ts_tablet_manager.cc:1918] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:04.201722 7616 log.cc:1198] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/3d4511e84bd7406cafe8691e0216ad93
I20250110 20:58:04.202467 7609 raft_consensus.cc:357] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.204146 7616 ts_tablet_manager.cc:1939] T 3d4511e84bd7406cafe8691e0216ad93 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:04.203694 7609 raft_consensus.cc:383] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.204597 7609 raft_consensus.cc:738] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.205269 7609 consensus_queue.cc:260] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.208444 7926 raft_consensus.cc:491] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.208861 7926 raft_consensus.cc:513] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.209846 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet f3e994b6ed3e4692a4ba3cfcf3b72648 (table default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]) successfully deleted
W20250110 20:58:04.210546 6979 catalog_manager.cc:4839] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): delete failed for tablet a580b2f85d854ff9979333b409bd6a77 because the tablet was not found. No further retry: Not found: Tablet not found: a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:04.209327 7609 ts_tablet_manager.cc:1428] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20250110 20:58:04.211530 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533494271bfa4d7486026a00767cf879" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:04.211994 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533494271bfa4d7486026a00767cf879" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:04.212121 7500 raft_consensus.cc:2463] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:04.212612 7210 raft_consensus.cc:2463] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:04.213878 7287 leader_election.cc:304] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:04.214609 8015 raft_consensus.cc:2798] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:04.214900 8015 raft_consensus.cc:491] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:04.215167 8015 raft_consensus.cc:3054] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.210637 7926 leader_election.cc:290] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.222419 8015 raft_consensus.cc:513] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.224830 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533494271bfa4d7486026a00767cf879" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:04.225344 7210 raft_consensus.cc:3054] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.232353 7210 raft_consensus.cc:2463] T 533494271bfa4d7486026a00767cf879 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:04.233093 8015 leader_election.cc:290] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.234131 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533494271bfa4d7486026a00767cf879" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:04.234274 7287 leader_election.cc:304] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:04.234685 7500 raft_consensus.cc:3054] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.235080 8015 raft_consensus.cc:2798] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:04.236689 8015 raft_consensus.cc:695] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:04.239298 8015 consensus_queue.cc:237] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.241693 7500 raft_consensus.cc:2463] T 533494271bfa4d7486026a00767cf879 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
W20250110 20:58:04.255048 6980 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.253:40203, user_credentials={real_user=slave}} blocked reactor thread for 57891.4us
I20250110 20:58:04.256626 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 3d4511e84bd7406cafe8691e0216ad93 (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]) successfully deleted
I20250110 20:58:04.256950 7911 raft_consensus.cc:491] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.257390 7911 raft_consensus.cc:513] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.259274 7911 leader_election.cc:290] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:04.263309 6990 catalog_manager.cc:5526] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:04.264515 6990 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:04.268491 7602 tablet_bootstrap.cc:492] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:04.268487 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c82412f7ba946b9b0dcffd8fa883353" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:04.269109 7353 raft_consensus.cc:2463] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
W20250110 20:58:04.269393 7435 leader_election.cc:343] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 1c82412f7ba946b9b0dcffd8fa883353
I20250110 20:58:04.270891 7436 leader_election.cc:304] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:58:04.271595 7911 raft_consensus.cc:2798] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:04.271939 7911 raft_consensus.cc:491] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:04.272249 7911 raft_consensus.cc:3054] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.273475 7602 tablet_bootstrap.cc:654] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.280910 7911 raft_consensus.cc:513] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.283283 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c82412f7ba946b9b0dcffd8fa883353" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
W20250110 20:58:04.284416 7435 leader_election.cc:343] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:04.286095 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c82412f7ba946b9b0dcffd8fa883353" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:04.286624 7353 raft_consensus.cc:3054] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.291163 6993 catalog_manager.cc:5526] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:04.300130 7353 raft_consensus.cc:2463] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:04.304960 7436 leader_election.cc:304] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:58:04.322055 7911 leader_election.cc:290] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
W20250110 20:58:04.324343 6990 catalog_manager.cc:5359] Ignoring report from unknown tablet ac84d3932d5e421fa0b641419eeebe6a
I20250110 20:58:04.324968 7911 raft_consensus.cc:2798] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:04.328847 6993 master_service.cc:946] ReplaceTablet: received request to replace tablet 12429ac2edaf49b9bdddb611e97ef1c3 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:04.338718 7911 raft_consensus.cc:695] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:58:04.339493 7911 consensus_queue.cc:237] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.351604 8015 raft_consensus.cc:491] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.352070 8015 raft_consensus.cc:513] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.353910 8015 leader_election.cc:290] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:04.355511 7288 leader_election.cc:343] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 43834def62374039abcbcab349edfe2e
W20250110 20:58:04.356127 7287 leader_election.cc:343] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 43834def62374039abcbcab349edfe2e
I20250110 20:58:04.356487 7287 leader_election.cc:304] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:04.357468 8015 raft_consensus.cc:2743] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:04.358410 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.359315 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.360152 7607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250110 20:58:04.362051 7478 tablet_service.cc:1467] Processing CreateTablet for tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.362989 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.363101 6990 catalog_manager.cc:5526] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:04.364569 6990 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:04.365929 7926 raft_consensus.cc:491] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.366346 7926 raft_consensus.cc:513] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.368039 7926 leader_election.cc:290] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:04.370085 7287 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:04.371876 7189 tablet_service.cc:1467] Processing CreateTablet for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (DEFAULT_TABLE table=default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.372840 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f58d4b7a1ff0439a9b1a08e9e04c044d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:58:04.373960 7288 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:04.373996 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8 (DEFAULT_TABLE table=default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.374292 7288 leader_election.cc:304] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:04.374857 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.375156 8015 raft_consensus.cc:2743] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:04.365175 6990 catalog_manager.cc:5526] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:04.378903 7333 tablet_service.cc:1467] Processing CreateTablet for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (DEFAULT_TABLE table=default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.379791 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f58d4b7a1ff0439a9b1a08e9e04c044d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.382009 7477 tablet_service.cc:1467] Processing CreateTablet for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (DEFAULT_TABLE table=default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.382920 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f58d4b7a1ff0439a9b1a08e9e04c044d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.393632 8039 consensus_queue.cc:1035] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.401721 7609 tablet_bootstrap.cc:492] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:04.401742 7606 tablet_bootstrap.cc:492] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:04.406635 7609 tablet_bootstrap.cc:654] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.406725 7606 tablet_bootstrap.cc:654] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.417577 7500 raft_consensus.cc:1270] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:04.418025 7210 raft_consensus.cc:1270] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer 9ef4ce3d44cc486fb0ee0ad85127011b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:04.419332 8015 consensus_queue.cc:1035] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.420580 8015 consensus_queue.cc:1035] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.421919 7609 tablet_bootstrap.cc:492] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:04.422354 7609 ts_tablet_manager.cc:1397] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.001s
I20250110 20:58:04.425035 7609 raft_consensus.cc:357] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.427891 6993 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 12429ac2edaf49b9bdddb611e97ef1c3
I20250110 20:58:04.429149 6993 catalog_manager.cc:6596] ReplaceTablet: tablet 12429ac2edaf49b9bdddb611e97ef1c3 deleted and replaced by tablet 9501a5e3c7354c45b8b09ef28e294dc1
I20250110 20:58:04.433657 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 12429ac2edaf49b9bdddb611e97ef1c3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9501a5e3c7354c45b8b09ef28e294dc1) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:04.434365 7615 tablet_replica.cc:331] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:04.436497 7617 ts_tablet_manager.cc:1918] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:04.436861 7617 log.cc:1198] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/77661375951e456899d2f9052dc680e7
I20250110 20:58:04.446426 7607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250110 20:58:04.446971 7607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250110 20:58:04.457739 7606 tablet_bootstrap.cc:492] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:04.459635 7606 ts_tablet_manager.cc:1397] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.001s
W20250110 20:58:04.464701 7210 raft_consensus.cc:1574] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 2' and following 0 ops. Status for this op: Illegal state: STOPPING
I20250110 20:58:04.469636 7615 raft_consensus.cc:2238] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.470338 7609 raft_consensus.cc:383] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.471123 7609 raft_consensus.cc:738] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.472162 7606 raft_consensus.cc:357] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:58:04.447655 7436 consensus_peers.cc:487] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a. This is attempt 6: this message will repeat every 5th retry.
I20250110 20:58:04.473240 7609 consensus_queue.cc:260] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.478438 7602 tablet_bootstrap.cc:492] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:04.478760 7606 raft_consensus.cc:383] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.478843 7602 ts_tablet_manager.cc:1397] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.211s user 0.049s sys 0.044s
I20250110 20:58:04.480697 7609 ts_tablet_manager.cc:1428] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.058s user 0.004s sys 0.000s
I20250110 20:58:04.480981 7606 raft_consensus.cc:738] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
W20250110 20:58:04.481130 6990 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 1c82412f7ba946b9b0dcffd8fa883353 consensus state has no leader
I20250110 20:58:04.476300 7617 ts_tablet_manager.cc:1939] T 77661375951e456899d2f9052dc680e7 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
W20250110 20:58:04.482129 7287 consensus_peers.cc:487] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Couldn't send request to peer 244d9ba396f549f2a5015792b6c5fff8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:58:04.482092 7606 consensus_queue.cc:260] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.489598 7609 tablet_bootstrap.cc:492] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:04.476697 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 12429ac2edaf49b9bdddb611e97ef1c3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9501a5e3c7354c45b8b09ef28e294dc1) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:04.491441 7606 ts_tablet_manager.cc:1428] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.028s user 0.003s sys 0.001s
I20250110 20:58:04.492249 7606 tablet_bootstrap.cc:492] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:04.484297 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 115) took 1672 ms. Trace:
I20250110 20:58:04.499805 7617 rpcz_store.cc:276] 0110 20:58:02.811665 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:02.811770 (+ 105us) service_pool.cc:225] Handling call
0110 20:58:04.049637 (+1237867us) ts_tablet_manager.cc:1145] Deleting tablet 77661375951e456899d2f9052dc680e7
0110 20:58:04.049680 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:04.323124 (+273444us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:04.436282 (+113158us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:04.484239 (+ 47957us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1237310}
I20250110 20:58:04.501044 7617 tablet_replica.cc:331] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:04.501828 7617 raft_consensus.cc:2238] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:04.504307 7617 raft_consensus.cc:2267] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.504772 7609 tablet_bootstrap.cc:654] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.502869 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet 77661375951e456899d2f9052dc680e7 (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]) successfully deleted
I20250110 20:58:04.507511 7615 pending_rounds.cc:62] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Trying to abort 1 pending ops.
I20250110 20:58:04.507748 7615 pending_rounds.cc:69] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878833200484352 op_type: NO_OP noop_request { }
I20250110 20:58:04.508072 7615 raft_consensus.cc:2883] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:04.508383 7615 raft_consensus.cc:2267] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.510536 7615 ts_tablet_manager.cc:1905] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.513026 7602 raft_consensus.cc:357] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.513741 7602 raft_consensus.cc:383] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.514019 7602 raft_consensus.cc:738] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.515012 7606 tablet_bootstrap.cc:654] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.515094 7602 consensus_queue.cc:260] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.517283 7602 ts_tablet_manager.cc:1428] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.038s user 0.003s sys 0.003s
I20250110 20:58:04.530629 7615 ts_tablet_manager.cc:1918] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:04.537200 7615 log.cc:1198] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/12429ac2edaf49b9bdddb611e97ef1c3
I20250110 20:58:04.537932 7615 ts_tablet_manager.cc:1939] T 12429ac2edaf49b9bdddb611e97ef1c3 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:04.538597 7609 tablet_bootstrap.cc:492] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:04.538978 7609 ts_tablet_manager.cc:1397] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.003s
I20250110 20:58:04.540149 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]) successfully deleted
I20250110 20:58:04.541456 6987 catalog_manager.cc:2149] created HMS catalog entry for table [id=c937038ba33645559692b4937a1fd7b6]
I20250110 20:58:04.540496 7602 tablet_bootstrap.cc:492] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
W20250110 20:58:04.455561 7435 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.171.193:36745, user_credentials={real_user=slave}} blocked reactor thread for 72320.4us
I20250110 20:58:04.541448 7609 raft_consensus.cc:357] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.559903 7609 raft_consensus.cc:383] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.547035 8017 raft_consensus.cc:491] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.560189 7609 raft_consensus.cc:738] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.560428 8017 raft_consensus.cc:513] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.560892 6987 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:33774 (request call id 89) took 2715 ms. Trace:
I20250110 20:58:04.560743 7609 consensus_queue.cc:260] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.562382 8017 leader_election.cc:290] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.561173 6987 rpcz_store.cc:276] 0110 20:58:01.845023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:01.849228 (+ 4205us) service_pool.cc:225] Handling call
0110 20:58:03.844699 (+1995471us) catalog_manager.cc:2076] Acquired catalog manager lock
0110 20:58:03.846947 (+ 2248us) catalog_manager.cc:2122] Created new table and tablet info
0110 20:58:04.541422 (+694475us) catalog_manager.cc:2148] Created new table in HMS catalog
0110 20:58:04.560279 (+ 18857us) catalog_manager.cc:2182] Wrote table and tablets to system table
0110 20:58:04.560589 (+ 310us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0110 20:58:04.560834 (+ 245us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0110 20:58:04.542971 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0110 20:58:04.543292 (+ 321us) write_op.cc:432] Acquiring schema lock in shared mode
0110 20:58:04.543314 (+ 22us) write_op.cc:435] Acquired schema lock
0110 20:58:04.543327 (+ 13us) tablet.cc:659] Decoding operations
0110 20:58:04.543564 (+ 237us) write_op.cc:620] Acquiring the partition lock for write op
0110 20:58:04.543601 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0110 20:58:04.543622 (+ 21us) tablet.cc:682] Acquiring locks for 2 operations
0110 20:58:04.543799 (+ 177us) tablet.cc:698] Row locks acquired
0110 20:58:04.543811 (+ 12us) write_op.cc:260] PREPARE: finished
0110 20:58:04.543925 (+ 114us) write_op.cc:270] Start()
0110 20:58:04.544003 (+ 78us) write_op.cc:276] Timestamp: P: 1736542684543841 usec, L: 0
0110 20:58:04.544017 (+ 14us) op_driver.cc:348] REPLICATION: starting
0110 20:58:04.544465 (+ 448us) log.cc:844] Serialized 433 byte log entry
0110 20:58:04.558261 (+ 13796us) op_driver.cc:464] REPLICATION: finished
0110 20:58:04.558544 (+ 283us) write_op.cc:301] APPLY: starting
0110 20:58:04.558594 (+ 50us) tablet.cc:1364] starting BulkCheckPresence
0110 20:58:04.558688 (+ 94us) tablet.cc:1367] finished BulkCheckPresence
0110 20:58:04.558776 (+ 88us) tablet.cc:1369] starting ApplyRowOperation cycle
0110 20:58:04.559137 (+ 361us) tablet.cc:1380] finished ApplyRowOperation cycle
0110 20:58:04.559265 (+ 128us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0110 20:58:04.559306 (+ 41us) write_op.cc:312] APPLY: finished
0110 20:58:04.559477 (+ 171us) log.cc:844] Serialized 29 byte log entry
0110 20:58:04.559610 (+ 133us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:04.559679 (+ 69us) write_op.cc:454] Released schema lock
0110 20:58:04.559710 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0110 20:58:04.559930 (+ 220us) write_op.cc:489] Releasing partition, row and schema locks
0110 20:58:04.559951 (+ 21us) write_op.cc:454] Released schema lock
Metrics: {"Hive Metastore.queue_time_us":1615,"Hive Metastore.run_cpu_time_us":594,"Hive Metastore.run_wall_time_us":689248,"child_traces":[["op",{"apply.queue_time_us":162,"apply.run_cpu_time_us":1332,"apply.run_wall_time_us":1332,"num_ops":2,"prepare.queue_time_us":103,"prepare.run_cpu_time_us":1727,"prepare.run_wall_time_us":1728,"raft.queue_time_us":1075,"raft.run_cpu_time_us":514,"raft.run_wall_time_us":514,"replication_time_us":14163}]]}
W20250110 20:58:04.564363 7287 leader_election.cc:343] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 8c518fd5cc6a4d1e8f7fc47489fe42a8
I20250110 20:58:04.536520 7617 ts_tablet_manager.cc:1905] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.566876 7609 ts_tablet_manager.cc:1428] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250110 20:58:04.577162 8039 consensus_queue.cc:1035] T fb8d5bf6a9694240b10e1b1c178b528b P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.585810 7602 tablet_bootstrap.cc:654] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.588054 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c518fd5cc6a4d1e8f7fc47489fe42a8" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:04.588588 7501 raft_consensus.cc:2463] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
W20250110 20:58:04.595443 7288 consensus_peers.cc:487] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Couldn't send request to peer c3b63d95ca734d8a9803b96bfd94f307. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f25437547b454eafbcf56e2b6c3b348a. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:58:04.596096 7288 leader_election.cc:304] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:58:04.599344 8017 raft_consensus.cc:2798] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:04.599665 8017 raft_consensus.cc:491] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:04.599973 8017 raft_consensus.cc:3054] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.613627 8017 raft_consensus.cc:513] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.616335 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c518fd5cc6a4d1e8f7fc47489fe42a8" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:04.616931 8017 leader_election.cc:290] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.620050 7477 tablet_service.cc:1514] Processing DeleteTablet for tablet 12429ac2edaf49b9bdddb611e97ef1c3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9501a5e3c7354c45b8b09ef28e294dc1) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:04.620790 7616 tablet_replica.cc:331] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
W20250110 20:58:04.622931 7287 leader_election.cc:343] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:04.626531 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c518fd5cc6a4d1e8f7fc47489fe42a8" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:04.627071 7501 raft_consensus.cc:3054] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.635506 7501 raft_consensus.cc:2463] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:04.636096 7602 tablet_bootstrap.cc:492] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:04.636508 7602 ts_tablet_manager.cc:1397] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.096s user 0.005s sys 0.004s
I20250110 20:58:04.650701 7602 raft_consensus.cc:357] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.651351 7602 raft_consensus.cc:383] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.651583 7602 raft_consensus.cc:738] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.660457 7288 leader_election.cc:304] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8
I20250110 20:58:04.662390 6993 catalog_manager.cc:5526] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:58:04.664155 6993 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:04.670818 7868 consensus_queue.cc:1035] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.681767 7602 consensus_queue.cc:260] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.683751 7602 ts_tablet_manager.cc:1428] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.040s user 0.003s sys 0.000s
I20250110 20:58:04.689616 6987 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:04.692570 6987 master_service.cc:946] ReplaceTablet: received request to replace tablet 18a3bdfc63a046f08ff4885e335c9758 from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:04.704579 7602 tablet_bootstrap.cc:492] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:04.714340 8039 raft_consensus.cc:491] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.714764 8039 raft_consensus.cc:513] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.716452 8039 leader_election.cc:290] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:04.732666 6993 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:04.733482 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d4b7a1ff0439a9b1a08e9e04c044d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
W20250110 20:58:04.735242 7435 leader_election.cc:343] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:04.725275 7602 tablet_bootstrap.cc:654] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.739672 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 9501a5e3c7354c45b8b09ef28e294dc1 (DEFAULT_TABLE table=default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.740586 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9501a5e3c7354c45b8b09ef28e294dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.741791 8017 raft_consensus.cc:2798] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:04.742419 8017 raft_consensus.cc:695] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:04.743145 8017 consensus_queue.cc:237] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.668421 7617 ts_tablet_manager.cc:1918] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:04.718300 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d4b7a1ff0439a9b1a08e9e04c044d" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:04.746057 7617 log.cc:1198] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/ac84d3932d5e421fa0b641419eeebe6a
I20250110 20:58:04.746924 7617 ts_tablet_manager.cc:1939] T ac84d3932d5e421fa0b641419eeebe6a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
W20250110 20:58:04.747532 7436 leader_election.cc:343] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:04.747912 7436 leader_election.cc:304] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:04.748836 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 117) took 1780 ms. Trace:
I20250110 20:58:04.748982 7616 raft_consensus.cc:2238] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.749172 7617 rpcz_store.cc:276] 0110 20:58:02.968013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:02.968139 (+ 126us) service_pool.cc:225] Handling call
0110 20:58:04.500908 (+1532769us) ts_tablet_manager.cc:1145] Deleting tablet ac84d3932d5e421fa0b641419eeebe6a
0110 20:58:04.500947 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:04.629469 (+128522us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:04.668208 (+ 38739us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:04.748784 (+ 80576us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532274}
I20250110 20:58:04.751344 8039 raft_consensus.cc:2743] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:04.751490 7911 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.751938 7911 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.753713 7911 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:58:04.754205 7616 pending_rounds.cc:62] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Trying to abort 2 pending ops.
I20250110 20:58:04.754410 7616 pending_rounds.cc:69] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7112878835361722368 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "12429ac2edaf49b9bdddb611e97ef1c3" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.table_176fb71364f24095a1efbf0d29c18ff8" dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" new_extra_config { } }
I20250110 20:58:04.755455 7616 pending_rounds.cc:69] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7112878833200484352 op_type: NO_OP noop_request { }
I20250110 20:58:04.755796 7616 raft_consensus.cc:2883] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250110 20:58:04.756088 7616 raft_consensus.cc:2267] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.758249 7616 ts_tablet_manager.cc:1905] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250110 20:58:04.759393 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:04.766101 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:04.766543 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:04.767218 7911 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:04.768337 7617 tablet_replica.cc:331] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:04.769068 7617 raft_consensus.cc:2238] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:04.769924 7617 raft_consensus.cc:2267] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.771562 7617 ts_tablet_manager.cc:1905] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.758378 8017 raft_consensus.cc:491] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.786623 7954 consensus_queue.cc:1035] T 47accefc003b4e2eae08990302350167 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.786867 8017 raft_consensus.cc:513] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.788841 7616 ts_tablet_manager.cc:1918] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:04.789198 7616 log.cc:1198] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/12429ac2edaf49b9bdddb611e97ef1c3
I20250110 20:58:04.789584 7606 tablet_bootstrap.cc:492] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:04.789852 7616 ts_tablet_manager.cc:1939] T 12429ac2edaf49b9bdddb611e97ef1c3 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:04.789981 7606 ts_tablet_manager.cc:1397] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.298s user 0.027s sys 0.030s
I20250110 20:58:04.790207 8017 leader_election.cc:290] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:04.791392 7287 leader_election.cc:343] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:04.792395 7606 raft_consensus.cc:357] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.792667 7477 tablet_service.cc:1467] Processing CreateTablet for tablet 9501a5e3c7354c45b8b09ef28e294dc1 (DEFAULT_TABLE table=default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.793015 7606 raft_consensus.cc:383] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:04.793236 7606 raft_consensus.cc:738] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:04.793721 7477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9501a5e3c7354c45b8b09ef28e294dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.793800 7606 consensus_queue.cc:260] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:04.795809 7606 ts_tablet_manager.cc:1428] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250110 20:58:04.737313 6990 master_service.cc:946] ReplaceTablet: received request to replace tablet 833cc8d477374f51a67422cbe051254e from {username='slave'} at 127.0.0.1:33774
I20250110 20:58:04.741166 6987 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 18a3bdfc63a046f08ff4885e335c9758
I20250110 20:58:04.802038 6987 catalog_manager.cc:6596] ReplaceTablet: tablet 18a3bdfc63a046f08ff4885e335c9758 deleted and replaced by tablet f08ccff4e5d84861b6dc6a6cb5142416
I20250110 20:58:04.792317 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 12429ac2edaf49b9bdddb611e97ef1c3 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]) successfully deleted
W20250110 20:58:04.750727 7288 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.195:45717, user_credentials={real_user=slave}} blocked reactor thread for 90352.1us
I20250110 20:58:04.805071 6980 catalog_manager.cc:4872] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): tablet ac84d3932d5e421fa0b641419eeebe6a (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]) successfully deleted
I20250110 20:58:04.805723 7911 raft_consensus.cc:491] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.806105 7911 raft_consensus.cc:513] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.807715 7911 leader_election.cc:290] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:04.808480 7926 raft_consensus.cc:491] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:04.808892 7926 raft_consensus.cc:513] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.810653 7926 leader_election.cc:290] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.813038 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:04.813652 7353 raft_consensus.cc:2463] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:04.813840 7333 tablet_service.cc:1467] Processing CreateTablet for tablet 9501a5e3c7354c45b8b09ef28e294dc1 (DEFAULT_TABLE table=default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.814756 7333 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9501a5e3c7354c45b8b09ef28e294dc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.814853 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:04.815382 7210 raft_consensus.cc:2463] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:04.816437 7332 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a3bdfc63a046f08ff4885e335c9758 with delete_type TABLET_DATA_DELETED (replaced by tablet f08ccff4e5d84861b6dc6a6cb5142416) from {username='slave'} at 127.0.0.1:40394
W20250110 20:58:04.816781 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:04.818305 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a3bdfc63a046f08ff4885e335c9758 with delete_type TABLET_DATA_DELETED (replaced by tablet f08ccff4e5d84861b6dc6a6cb5142416) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:04.818897 7616 tablet_replica.cc:331] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:04.819590 7616 raft_consensus.cc:2238] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.819995 7616 raft_consensus.cc:2267] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.820578 7287 leader_election.cc:304] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:04.822263 7616 ts_tablet_manager.cc:1905] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.822988 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:04.823558 7501 raft_consensus.cc:2463] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:04.821928 7926 raft_consensus.cc:2798] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:04.824122 7926 raft_consensus.cc:491] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:04.824432 7926 raft_consensus.cc:3054] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.831056 7436 leader_election.cc:304] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:04.834347 7606 tablet_bootstrap.cc:492] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
W20250110 20:58:04.835413 7288 leader_election.cc:343] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:04.835783 7288 leader_election.cc:304] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:04.835835 8066 tablet.cc:1721] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:04.836409 8015 raft_consensus.cc:2743] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:04.837491 7911 raft_consensus.cc:2798] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:04.837802 7911 raft_consensus.cc:491] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:04.838095 7911 raft_consensus.cc:3054] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.839876 7606 tablet_bootstrap.cc:654] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:04.842604 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:04.843210 7210 raft_consensus.cc:2463] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:04.850607 7926 raft_consensus.cc:513] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.863070 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:04.863626 7210 raft_consensus.cc:3054] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.864619 7616 ts_tablet_manager.cc:1918] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:04.865057 7616 log.cc:1198] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/18a3bdfc63a046f08ff4885e335c9758
I20250110 20:58:04.865727 7616 ts_tablet_manager.cc:1939] T 18a3bdfc63a046f08ff4885e335c9758 P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:04.865836 7926 leader_election.cc:290] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:04.874619 6980 catalog_manager.cc:4872] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): tablet 18a3bdfc63a046f08ff4885e335c9758 (table default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]) successfully deleted
I20250110 20:58:04.880607 7209 raft_consensus.cc:3054] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:04.887434 7351 raft_consensus.cc:1270] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Refusing update from remote peer c3b63d95ca734d8a9803b96bfd94f307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:04.890328 7911 raft_consensus.cc:513] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.892066 7911 leader_election.cc:290] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:04.893399 7209 raft_consensus.cc:1270] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Refusing update from remote peer c3b63d95ca734d8a9803b96bfd94f307: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250110 20:58:04.895843 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:04.896499 7353 raft_consensus.cc:2388] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c3b63d95ca734d8a9803b96bfd94f307 in current term 1: Already voted for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in this term.
I20250110 20:58:04.893357 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:04.899274 7501 raft_consensus.cc:2388] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9ef4ce3d44cc486fb0ee0ad85127011b in current term 1: Already voted for candidate c3b63d95ca734d8a9803b96bfd94f307 in this term.
W20250110 20:58:04.894639 7436 consensus_peers.cc:487] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5. This is attempt 6: this message will repeat every 5th retry.
I20250110 20:58:04.894938 8039 consensus_queue.cc:1035] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.900663 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab2f24f374cb4dfba6d9563cb6666f6c" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:04.901644 8039 consensus_queue.cc:1035] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.904080 7435 leader_election.cc:304] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:04.904743 8039 raft_consensus.cc:2743] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250110 20:58:04.911285 8017 consensus_queue.cc:1035] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:04.927053 7189 tablet_service.cc:1467] Processing CreateTablet for tablet ac7a9bac7d9c434e9a8b3b9d5b67f4a5 (DEFAULT_TABLE table=default.table_eed9317ae3914199a2cdc0d7e112371f [id=c937038ba33645559692b4937a1fd7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.928015 7189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac7a9bac7d9c434e9a8b3b9d5b67f4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.930830 6987 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 1 got 0
I20250110 20:58:04.931835 6987 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:04.949942 6990 catalog_manager.cc:5813] T 00000000000000000000000000000000 P a36f292a1a20461e9aef5aefe8fd0f08: Sending DeleteTablet for 3 replicas of tablet 833cc8d477374f51a67422cbe051254e
I20250110 20:58:04.951128 6990 catalog_manager.cc:6596] ReplaceTablet: tablet 833cc8d477374f51a67422cbe051254e deleted and replaced by tablet 49d26618d4ee4608a9a0333929b654f4
I20250110 20:58:04.951197 7602 tablet_bootstrap.cc:492] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:04.951596 7602 ts_tablet_manager.cc:1397] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.247s user 0.008s sys 0.000s
I20250110 20:58:04.953234 7190 tablet_service.cc:1514] Processing DeleteTablet for tablet 833cc8d477374f51a67422cbe051254e with delete_type TABLET_DATA_DELETED (replaced by tablet 49d26618d4ee4608a9a0333929b654f4) from {username='slave'} at 127.0.0.1:42840
I20250110 20:58:04.953869 7615 tablet_replica.cc:331] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:04.953771 6990 catalog_manager.cc:1909] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33774:
name: "default.Table_7d7e944c87674880ad7f3b8a5a5460ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250110 20:58:04.954521 7615 raft_consensus.cc:2238] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.954936 7615 raft_consensus.cc:2267] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.955638 7478 tablet_service.cc:1514] Processing DeleteTablet for tablet 833cc8d477374f51a67422cbe051254e with delete_type TABLET_DATA_DELETED (replaced by tablet 49d26618d4ee4608a9a0333929b654f4) from {username='slave'} at 127.0.0.1:43396
I20250110 20:58:04.956285 7616 tablet_replica.cc:331] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: stopping tablet replica
I20250110 20:58:04.956537 7615 ts_tablet_manager.cc:1905] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.957013 7616 raft_consensus.cc:2238] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:04.957545 7616 raft_consensus.cc:2267] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:04.959059 7616 ts_tablet_manager.cc:1905] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:04.963040 7210 raft_consensus.cc:2463] T ab2f24f374cb4dfba6d9563cb6666f6c P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:04.964048 7287 leader_election.cc:304] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:04.964851 8017 raft_consensus.cc:2798] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:04.965526 8017 raft_consensus.cc:695] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:04.966328 8017 consensus_queue.cc:237] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:04.969374 7188 tablet_service.cc:1514] Processing DeleteTablet for tablet 18a3bdfc63a046f08ff4885e335c9758 with delete_type TABLET_DATA_DELETED (replaced by tablet f08ccff4e5d84861b6dc6a6cb5142416) from {username='slave'} at 127.0.0.1:42840
W20250110 20:58:04.972282 7288 consensus_peers.cc:487] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Couldn't send request to peer c3b63d95ca734d8a9803b96bfd94f307. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12429ac2edaf49b9bdddb611e97ef1c3. This is attempt 1: this message will repeat every 5th retry.
W20250110 20:58:04.973625 7288 consensus_peers.cc:487] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b -> Peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Couldn't send request to peer c3b63d95ca734d8a9803b96bfd94f307. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 18a3bdfc63a046f08ff4885e335c9758. This is attempt 1: this message will repeat every 5th retry.
I20250110 20:58:04.974375 7333 tablet_service.cc:1514] Processing DeleteTablet for tablet 833cc8d477374f51a67422cbe051254e with delete_type TABLET_DATA_DELETED (replaced by tablet 49d26618d4ee4608a9a0333929b654f4) from {username='slave'} at 127.0.0.1:40394
I20250110 20:58:04.974234 7478 tablet_service.cc:1467] Processing CreateTablet for tablet ac7a9bac7d9c434e9a8b3b9d5b67f4a5 (DEFAULT_TABLE table=default.table_eed9317ae3914199a2cdc0d7e112371f [id=c937038ba33645559692b4937a1fd7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.975199 7478 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac7a9bac7d9c434e9a8b3b9d5b67f4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.975312 7332 tablet_service.cc:1467] Processing CreateTablet for tablet ac7a9bac7d9c434e9a8b3b9d5b67f4a5 (DEFAULT_TABLE table=default.table_eed9317ae3914199a2cdc0d7e112371f [id=c937038ba33645559692b4937a1fd7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.976188 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac7a9bac7d9c434e9a8b3b9d5b67f4a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.977984 7331 tablet_service.cc:1467] Processing CreateTablet for tablet f08ccff4e5d84861b6dc6a6cb5142416 (DEFAULT_TABLE table=default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.978425 7188 tablet_service.cc:1467] Processing CreateTablet for tablet f08ccff4e5d84861b6dc6a6cb5142416 (DEFAULT_TABLE table=default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.978829 7331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08ccff4e5d84861b6dc6a6cb5142416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.979308 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08ccff4e5d84861b6dc6a6cb5142416. 1 dirs total, 0 dirs full, 0 dirs failed
W20250110 20:58:04.981423 6987 catalog_manager.cc:5718] Failed to send AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
I20250110 20:58:04.983024 7481 tablet_service.cc:1514] Processing DeleteTablet for tablet 12429ac2edaf49b9bdddb611e97ef1c3 with delete_type TABLET_DATA_DELETED (replaced by tablet 9501a5e3c7354c45b8b09ef28e294dc1) from {username='slave'} at 127.0.0.1:43396
W20250110 20:58:04.990931 6993 catalog_manager.cc:5359] Ignoring report from unknown tablet f25437547b454eafbcf56e2b6c3b348a
I20250110 20:58:04.998051 7481 tablet_service.cc:1467] Processing CreateTablet for tablet f08ccff4e5d84861b6dc6a6cb5142416 (DEFAULT_TABLE table=default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:04.999040 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f08ccff4e5d84861b6dc6a6cb5142416. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:04.999511 8066 tablet.cc:1721] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:05.001827 7602 raft_consensus.cc:357] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.002480 7602 raft_consensus.cc:383] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.002704 7602 raft_consensus.cc:738] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.003202 7602 consensus_queue.cc:260] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:58:05.007858 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
I20250110 20:58:05.008946 7609 tablet_bootstrap.cc:492] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:05.013864 7609 tablet_bootstrap.cc:654] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.018110 7615 ts_tablet_manager.cc:1918] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:05.018457 7615 log.cc:1198] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/833cc8d477374f51a67422cbe051254e
I20250110 20:58:05.019055 7615 ts_tablet_manager.cc:1939] T 833cc8d477374f51a67422cbe051254e P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:05.021447 7602 ts_tablet_manager.cc:1428] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.070s user 0.005s sys 0.001s
I20250110 20:58:05.022207 7602 tablet_bootstrap.cc:492] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
W20250110 20:58:05.023387 6987 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:05.031216 7602 tablet_bootstrap.cc:654] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.034075 7606 tablet_bootstrap.cc:492] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:05.035032 7606 ts_tablet_manager.cc:1397] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.201s user 0.046s sys 0.024s
I20250110 20:58:04.992940 6993 catalog_manager.cc:5526] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
I20250110 20:58:05.036633 6993 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 1 got 0
I20250110 20:58:05.037166 7606 raft_consensus.cc:357] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.038765 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 833cc8d477374f51a67422cbe051254e (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]) successfully deleted
I20250110 20:58:05.039113 7606 raft_consensus.cc:383] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.039359 7606 raft_consensus.cc:738] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.039922 7606 consensus_queue.cc:260] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.041872 7606 ts_tablet_manager.cc:1428] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250110 20:58:05.042440 7615 tablet_replica.cc:331] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: stopping tablet replica
I20250110 20:58:05.043148 7615 raft_consensus.cc:2238] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:05.043573 7615 raft_consensus.cc:2267] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.049069 7606 tablet_bootstrap.cc:492] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:05.049818 8077 tablet.cc:1721] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:05.055313 7615 ts_tablet_manager.cc:1905] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.070379 7606 tablet_bootstrap.cc:654] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.078644 8079 tablet.cc:1721] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:05.086087 7609 tablet_bootstrap.cc:492] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:05.086557 7609 ts_tablet_manager.cc:1397] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.078s user 0.018s sys 0.004s
I20250110 20:58:05.089138 7609 raft_consensus.cc:357] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.089847 7609 raft_consensus.cc:383] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.090185 7609 raft_consensus.cc:738] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.091012 7609 consensus_queue.cc:260] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.094569 7602 tablet_bootstrap.cc:492] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:05.094957 7602 ts_tablet_manager.cc:1397] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.073s user 0.007s sys 0.002s
I20250110 20:58:05.096863 7609 ts_tablet_manager.cc:1428] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20250110 20:58:05.097673 7609 tablet_bootstrap.cc:492] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:05.097126 7602 raft_consensus.cc:357] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.098104 7602 raft_consensus.cc:383] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.098352 7602 raft_consensus.cc:738] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.098950 7602 consensus_queue.cc:260] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.104737 7602 ts_tablet_manager.cc:1428] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20250110 20:58:05.108024 7609 tablet_bootstrap.cc:654] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.113732 7606 tablet_bootstrap.cc:492] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:05.114125 7606 ts_tablet_manager.cc:1397] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.001s
I20250110 20:58:05.109371 7616 ts_tablet_manager.cc:1918] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:05.118767 7609 tablet_bootstrap.cc:492] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:05.119254 7609 ts_tablet_manager.cc:1397] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.012s
I20250110 20:58:05.121516 7609 raft_consensus.cc:357] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.122474 7609 raft_consensus.cc:383] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.122757 7609 raft_consensus.cc:738] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.118789 7616 log.cc:1198] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-2/wal/wals/833cc8d477374f51a67422cbe051254e
I20250110 20:58:05.123745 7616 ts_tablet_manager.cc:1939] T 833cc8d477374f51a67422cbe051254e P c3b63d95ca734d8a9803b96bfd94f307: Deleting consensus metadata
I20250110 20:58:05.125995 7609 consensus_queue.cc:260] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.126827 7602 tablet_bootstrap.cc:492] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:05.121268 7617 ts_tablet_manager.cc:1918] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:05.127378 7617 log.cc:1198] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f25437547b454eafbcf56e2b6c3b348a
I20250110 20:58:05.128019 7617 ts_tablet_manager.cc:1939] T f25437547b454eafbcf56e2b6c3b348a P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.132543 7609 ts_tablet_manager.cc:1428] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.013s user 0.000s sys 0.004s
I20250110 20:58:05.121301 7606 raft_consensus.cc:357] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.134938 7606 raft_consensus.cc:383] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.135025 7609 tablet_bootstrap.cc:492] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:05.135186 7606 raft_consensus.cc:738] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.137053 7615 ts_tablet_manager.cc:1918] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:05.137790 7606 consensus_queue.cc:260] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.138657 7615 log.cc:1198] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-0/wal/wals/18a3bdfc63a046f08ff4885e335c9758
I20250110 20:58:05.139343 7615 ts_tablet_manager.cc:1939] T 18a3bdfc63a046f08ff4885e335c9758 P 244d9ba396f549f2a5015792b6c5fff8: Deleting consensus metadata
I20250110 20:58:05.140585 7609 tablet_bootstrap.cc:654] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.142702 7911 raft_consensus.cc:491] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:05.143107 7911 raft_consensus.cc:513] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.144817 7911 leader_election.cc:290] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:05.146984 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7a9bac7d9c434e9a8b3b9d5b67f4a5" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:05.148029 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7a9bac7d9c434e9a8b3b9d5b67f4a5" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:05.149312 7602 tablet_bootstrap.cc:654] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
W20250110 20:58:05.149269 7436 leader_election.cc:343] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Illegal state: must be running to vote when last-logged opid is not known
W20250110 20:58:05.150218 7435 leader_election.cc:343] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:05.150580 7435 leader_election.cc:304] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:05.151546 7911 raft_consensus.cc:2743] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:05.152122 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 118) took 2105 ms. Trace:
I20250110 20:58:05.152434 7617 rpcz_store.cc:276] 0110 20:58:03.047079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:03.047224 (+ 145us) service_pool.cc:225] Handling call
0110 20:58:04.768187 (+1720963us) ts_tablet_manager.cc:1145] Deleting tablet f25437547b454eafbcf56e2b6c3b348a
0110 20:58:04.768239 (+ 52us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:05.035722 (+267483us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.121057 (+ 85335us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.152067 (+ 31010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1720437}
I20250110 20:58:05.153535 7617 tablet_replica.cc:331] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:05.154196 7617 raft_consensus.cc:2238] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:05.154723 7617 raft_consensus.cc:2267] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.156301 7617 ts_tablet_manager.cc:1905] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.159646 8017 raft_consensus.cc:491] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:05.160063 8017 raft_consensus.cc:513] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.162468 8017 leader_election.cc:290] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:05.164667 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ccff4e5d84861b6dc6a6cb5142416" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
W20250110 20:58:05.166622 7287 leader_election.cc:343] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:05.167131 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ccff4e5d84861b6dc6a6cb5142416" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
W20250110 20:58:05.168257 7288 leader_election.cc:343] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Illegal state: must be running to vote when last-logged opid is not known
I20250110 20:58:05.168591 7288 leader_election.cc:304] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:05.169186 8017 raft_consensus.cc:2743] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:05.167685 7606 ts_tablet_manager.cc:1428] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.053s user 0.004s sys 0.002s
I20250110 20:58:05.175899 7606 tablet_bootstrap.cc:492] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:05.179280 6979 catalog_manager.cc:4872] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): tablet 18a3bdfc63a046f08ff4885e335c9758 (table default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]) successfully deleted
I20250110 20:58:05.184202 7606 tablet_bootstrap.cc:654] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.196434 6992 catalog_manager.cc:3422] renamed table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401] in HMS: new name default.table_c3ed0987ec6e41d39ddc47cba082a635
I20250110 20:58:05.207513 7602 tablet_bootstrap.cc:492] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:05.210680 7606 tablet_bootstrap.cc:492] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:05.211154 7606 ts_tablet_manager.cc:1397] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.004s
I20250110 20:58:05.211941 7602 ts_tablet_manager.cc:1397] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.106s user 0.013s sys 0.000s
I20250110 20:58:05.214488 7609 tablet_bootstrap.cc:492] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:05.213390 7606 raft_consensus.cc:357] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.214893 7609 ts_tablet_manager.cc:1397] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.081s user 0.017s sys 0.041s
I20250110 20:58:05.215201 7606 raft_consensus.cc:383] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.215497 7606 raft_consensus.cc:738] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.215442 7602 raft_consensus.cc:357] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.216068 7602 raft_consensus.cc:383] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.216316 7602 raft_consensus.cc:738] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.217700 7606 consensus_queue.cc:260] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.219702 7617 ts_tablet_manager.cc:1918] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:05.219686 7609 raft_consensus.cc:357] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.220332 7609 raft_consensus.cc:383] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.220562 7609 raft_consensus.cc:738] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.221133 7609 consensus_queue.cc:260] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.227039 7609 ts_tablet_manager.cc:1428] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20250110 20:58:05.216854 7602 consensus_queue.cc:260] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.220027 7617 log.cc:1198] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/43834def62374039abcbcab349edfe2e
I20250110 20:58:05.241577 7602 ts_tablet_manager.cc:1428] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20250110 20:58:05.243590 7606 ts_tablet_manager.cc:1428] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.032s user 0.002s sys 0.003s
I20250110 20:58:05.247188 7617 ts_tablet_manager.cc:1939] T 43834def62374039abcbcab349edfe2e P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.254942 7602 tablet_bootstrap.cc:492] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:05.256363 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 120) took 1992 ms. Trace:
I20250110 20:58:05.256659 7617 rpcz_store.cc:276] 0110 20:58:03.263793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:03.264708 (+ 915us) service_pool.cc:225] Handling call
0110 20:58:05.153378 (+1888670us) ts_tablet_manager.cc:1145] Deleting tablet 43834def62374039abcbcab349edfe2e
0110 20:58:05.153437 (+ 59us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:05.195492 (+ 42055us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.219494 (+ 24002us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.256305 (+ 36811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1888154}
I20250110 20:58:05.265509 8017 raft_consensus.cc:491] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:05.266399 7617 tablet_replica.cc:331] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:05.266005 8017 raft_consensus.cc:513] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.267113 7617 raft_consensus.cc:2238] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:05.267527 7617 raft_consensus.cc:2267] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.267870 8017 leader_election.cc:290] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:05.269143 7617 ts_tablet_manager.cc:1905] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.270525 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d4b7a1ff0439a9b1a08e9e04c044d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:05.270566 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d4b7a1ff0439a9b1a08e9e04c044d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:05.271085 7208 raft_consensus.cc:2463] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:05.271072 7501 raft_consensus.cc:2463] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:05.272262 7288 leader_election.cc:304] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:05.274053 8017 raft_consensus.cc:2798] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:05.274451 8017 raft_consensus.cc:491] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:05.274811 8017 raft_consensus.cc:3054] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.281014 8017 raft_consensus.cc:513] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.286623 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d4b7a1ff0439a9b1a08e9e04c044d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:05.287134 7208 raft_consensus.cc:3054] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.287356 7602 tablet_bootstrap.cc:654] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.302546 7208 raft_consensus.cc:2463] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:05.304955 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f58d4b7a1ff0439a9b1a08e9e04c044d" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:05.305480 7501 raft_consensus.cc:3054] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.307114 8017 leader_election.cc:290] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:05.309754 7287 leader_election.cc:304] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:05.311472 8017 raft_consensus.cc:2798] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:05.313210 8017 raft_consensus.cc:695] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:05.314472 8017 consensus_queue.cc:237] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.321134 7501 raft_consensus.cc:2463] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:05.329041 7602 tablet_bootstrap.cc:492] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:05.329465 7602 ts_tablet_manager.cc:1397] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.077s user 0.008s sys 0.007s
I20250110 20:58:05.331506 7602 raft_consensus.cc:357] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.332149 7602 raft_consensus.cc:383] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.332386 7602 raft_consensus.cc:738] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.336246 7602 consensus_queue.cc:260] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.342890 7617 ts_tablet_manager.cc:1918] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:05.343258 7617 log.cc:1198] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/a580b2f85d854ff9979333b409bd6a77
I20250110 20:58:05.358449 7602 ts_tablet_manager.cc:1428] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20250110 20:58:05.365445 7602 tablet_bootstrap.cc:492] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:05.370721 7617 ts_tablet_manager.cc:1939] T a580b2f85d854ff9979333b409bd6a77 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.371865 8017 raft_consensus.cc:491] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:05.372320 8017 raft_consensus.cc:513] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.374152 8017 leader_election.cc:290] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:05.374786 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501a5e3c7354c45b8b09ef28e294dc1" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:05.375334 7208 raft_consensus.cc:2463] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:05.376533 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501a5e3c7354c45b8b09ef28e294dc1" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:05.376892 7287 leader_election.cc:304] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:05.377070 7501 raft_consensus.cc:2463] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:05.377632 8017 raft_consensus.cc:2798] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:05.377935 8017 raft_consensus.cc:491] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:05.378222 8017 raft_consensus.cc:3054] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.378604 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 123) took 1771 ms. Trace:
I20250110 20:58:05.378870 7617 rpcz_store.cc:276] 0110 20:58:03.607646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:03.607745 (+ 99us) service_pool.cc:225] Handling call
0110 20:58:05.266259 (+1658514us) ts_tablet_manager.cc:1145] Deleting tablet a580b2f85d854ff9979333b409bd6a77
0110 20:58:05.266298 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:05.292307 (+ 26009us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.342663 (+ 50356us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.378546 (+ 35883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1658008}
I20250110 20:58:05.385509 7617 tablet_replica.cc:331] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:05.385823 8017 raft_consensus.cc:513] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.390753 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501a5e3c7354c45b8b09ef28e294dc1" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:05.391247 7208 raft_consensus.cc:3054] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.396991 7208 raft_consensus.cc:2463] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:05.398166 7287 leader_election.cc:304] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:05.399008 8015 raft_consensus.cc:2798] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:05.399421 8015 raft_consensus.cc:695] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:05.400177 8015 consensus_queue.cc:237] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.407604 8017 leader_election.cc:290] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:05.407837 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9501a5e3c7354c45b8b09ef28e294dc1" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:05.408360 7501 raft_consensus.cc:3054] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.411032 7602 tablet_bootstrap.cc:654] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:05.386235 7617 raft_consensus.cc:2238] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:05.413338 7617 raft_consensus.cc:2267] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.414587 7501 raft_consensus.cc:2463] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:05.417201 7617 ts_tablet_manager.cc:1905] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.444121 8015 consensus_queue.cc:1035] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:05.454108 7602 tablet_bootstrap.cc:492] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:05.454505 7602 ts_tablet_manager.cc:1397] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.090s user 0.010s sys 0.002s
I20250110 20:58:05.456663 7602 raft_consensus.cc:357] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.466048 8015 consensus_queue.cc:1035] T 533494271bfa4d7486026a00767cf879 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:05.457295 7602 raft_consensus.cc:383] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:05.467571 7602 raft_consensus.cc:738] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
I20250110 20:58:05.474629 7602 consensus_queue.cc:260] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:05.491712 7602 ts_tablet_manager.cc:1428] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
I20250110 20:58:05.513140 6988 catalog_manager.cc:3422] renamed table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047] in HMS: new name default.table_58cf4fbf7d1a441ebe2a9611029cf17b
I20250110 20:58:05.529181 8116 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:58:05.533821 8116 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:05.536144 7617 ts_tablet_manager.cc:1918] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:05.536523 7617 log.cc:1198] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/f3e994b6ed3e4692a4ba3cfcf3b72648
I20250110 20:58:05.537226 7617 ts_tablet_manager.cc:1939] T f3e994b6ed3e4692a4ba3cfcf3b72648 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.551334 8116 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:05.553637 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 128) took 1476 ms. Trace:
I20250110 20:58:05.556325 7208 raft_consensus.cc:3054] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.563562 8015 consensus_queue.cc:1035] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:05.553952 7617 rpcz_store.cc:276] 0110 20:58:04.077558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:04.077670 (+ 112us) service_pool.cc:225] Handling call
0110 20:58:05.385348 (+1307678us) ts_tablet_manager.cc:1145] Deleting tablet f3e994b6ed3e4692a4ba3cfcf3b72648
0110 20:58:05.385387 (+ 39us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:05.453596 (+ 68209us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.535927 (+ 82331us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.553573 (+ 17646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1307146}
I20250110 20:58:05.570212 7617 tablet_replica.cc:331] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:05.571024 7617 raft_consensus.cc:2238] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:05.571519 7617 raft_consensus.cc:2267] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.583109 7617 ts_tablet_manager.cc:1905] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.594990 7926 consensus_queue.cc:1035] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:58:05.597266 6828 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111) [suppressed 12 similar messages]
I20250110 20:58:05.624053 6985 catalog_manager.cc:2149] created HMS catalog entry for table [id=0fa12d64926f4e499c814889d85e6a39]
W20250110 20:58:05.631033 6828 leader_election.cc:336] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203): Network error: Client connection negotiation failed: client connection to 127.4.171.253:40203: connect: Connection refused (error 111)
I20250110 20:58:05.670598 7617 ts_tablet_manager.cc:1918] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:05.671005 7617 log.cc:1198] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/3d4511e84bd7406cafe8691e0216ad93
I20250110 20:58:05.671662 7617 ts_tablet_manager.cc:1939] T 3d4511e84bd7406cafe8691e0216ad93 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.673741 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 129) took 1563 ms. Trace:
I20250110 20:58:05.674041 7617 rpcz_store.cc:276] 0110 20:58:04.111527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:04.111633 (+ 106us) service_pool.cc:225] Handling call
0110 20:58:05.570070 (+1458437us) ts_tablet_manager.cc:1145] Deleting tablet 3d4511e84bd7406cafe8691e0216ad93
0110 20:58:05.570110 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:05.650111 (+ 80001us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.670323 (+ 20212us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.673681 (+ 3358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1457879}
I20250110 20:58:05.675082 7617 tablet_replica.cc:331] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:05.675791 7617 raft_consensus.cc:2238] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:05.676712 7617 raft_consensus.cc:2267] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.681241 7617 ts_tablet_manager.cc:1905] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.708561 7617 ts_tablet_manager.cc:1918] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250110 20:58:05.708910 7617 log.cc:1198] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/12429ac2edaf49b9bdddb611e97ef1c3
I20250110 20:58:05.718428 7617 ts_tablet_manager.cc:1939] T 12429ac2edaf49b9bdddb611e97ef1c3 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.719012 6986 catalog_manager.cc:2149] created HMS catalog entry for table [id=b25bd85bc96a4460910a5c35147ac89b]
I20250110 20:58:05.724798 7617 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:40394 (request call id 133) took 1289 ms. Trace:
I20250110 20:58:05.728224 8138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.4.171.253:40203 every 8 connection(s)
I20250110 20:58:05.728783 7607 rpc_server.cc:307] RPC server started. Bound to: 127.4.171.253:40203
I20250110 20:58:05.732693 7617 rpcz_store.cc:276] 0110 20:58:04.434944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:04.476544 (+ 41600us) service_pool.cc:225] Handling call
0110 20:58:05.674941 (+1198397us) ts_tablet_manager.cc:1145] Deleting tablet 12429ac2edaf49b9bdddb611e97ef1c3
0110 20:58:05.674981 (+ 40us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0110 20:58:05.697930 (+ 22949us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.708348 (+ 10418us) tablet_metadata.cc:641] Metadata flushed
0110 20:58:05.724738 (+ 16390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1184556}
I20250110 20:58:05.734367 7617 tablet_replica.cc:331] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:05.735076 7617 raft_consensus.cc:2238] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Raft consensus shutting down.
I20250110 20:58:05.736524 7607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/master-1/data/info.pb
I20250110 20:58:05.738725 4783 external_mini_cluster.cc:1413] Started /tmp/dist-test-task1Eq_p2/build/tsan/bin/kudu as pid 7607
I20250110 20:58:05.735945 7617 raft_consensus.cc:2267] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.745621 7617 ts_tablet_manager.cc:1905] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.766801 8139 sys_catalog.cc:263] Verifying existing consensus state
I20250110 20:58:05.774915 8139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: Bootstrap starting.
I20250110 20:58:05.788986 7617 ts_tablet_manager.cc:1918] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250110 20:58:05.789342 7617 log.cc:1198] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/18a3bdfc63a046f08ff4885e335c9758
I20250110 20:58:05.790040 7617 ts_tablet_manager.cc:1939] T 18a3bdfc63a046f08ff4885e335c9758 P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.792189 7617 tablet_replica.cc:331] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: stopping tablet replica
I20250110 20:58:05.792857 7617 raft_consensus.cc:2238] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus shutting down.
I20250110 20:58:05.793259 7617 raft_consensus.cc:2267] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Raft consensus is shut down!
I20250110 20:58:05.794804 7617 ts_tablet_manager.cc:1905] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250110 20:58:05.797065 8139 log.cc:826] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: Log is configured to *not* fsync() on all Append() calls
I20250110 20:58:05.806655 7547 heartbeater.cc:346] Connected to a master server at 127.4.171.253:40203
I20250110 20:58:05.826845 7617 ts_tablet_manager.cc:1918] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250110 20:58:05.827198 7617 log.cc:1198] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting WAL directory at /tmp/dist-test-task1Eq_p2/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1736542608734198-4783-0/minicluster-data/ts-1/wal/wals/833cc8d477374f51a67422cbe051254e
I20250110 20:58:05.827783 7617 ts_tablet_manager.cc:1939] T 833cc8d477374f51a67422cbe051254e P 9ef4ce3d44cc486fb0ee0ad85127011b: Deleting consensus metadata
I20250110 20:58:05.838887 8139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250110 20:58:05.840116 8139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: Bootstrap complete.
I20250110 20:58:05.861661 7926 consensus_queue.cc:1035] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:05.878448 7256 heartbeater.cc:346] Connected to a master server at 127.4.171.253:40203
I20250110 20:58:05.905803 8015 raft_consensus.cc:491] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:05.906258 8015 raft_consensus.cc:513] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.908078 8015 leader_election.cc:290] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:05.913970 7210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7a9bac7d9c434e9a8b3b9d5b67f4a5" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:05.914523 7210 raft_consensus.cc:2463] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:05.916101 8017 consensus_queue.cc:1035] T ab2f24f374cb4dfba6d9563cb6666f6c P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:05.917135 7287 leader_election.cc:304] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:05.918023 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7a9bac7d9c434e9a8b3b9d5b67f4a5" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:05.918558 7501 raft_consensus.cc:2463] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 0.
I20250110 20:58:05.921260 8017 raft_consensus.cc:2798] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:05.921604 8017 raft_consensus.cc:491] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:05.921882 8017 raft_consensus.cc:3054] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.938395 8017 raft_consensus.cc:513] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.941205 7399 heartbeater.cc:346] Connected to a master server at 127.4.171.253:40203
I20250110 20:58:05.960367 7500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7a9bac7d9c434e9a8b3b9d5b67f4a5" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:05.960752 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7a9bac7d9c434e9a8b3b9d5b67f4a5" candidate_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:05.960964 7500 raft_consensus.cc:3054] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.961257 7208 raft_consensus.cc:3054] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:05.966847 7208 raft_consensus.cc:2463] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:05.967778 7287 leader_election.cc:304] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:05.968513 8015 raft_consensus.cc:2798] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:05.968914 8015 raft_consensus.cc:695] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 LEADER]: Becoming Leader. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Running, Role: LEADER
I20250110 20:58:05.970474 8015 consensus_queue.cc:237] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } }
I20250110 20:58:05.967895 7500 raft_consensus.cc:2463] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ef4ce3d44cc486fb0ee0ad85127011b in term 1.
I20250110 20:58:05.984154 8017 leader_election.cc:290] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:06.002693 8139 raft_consensus.cc:357] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:06.011457 8139 raft_consensus.cc:738] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9bb61bdf30044a1b482d6a66e29435f, State: Initialized, Role: FOLLOWER
I20250110 20:58:06.029625 8139 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:06.057557 8148 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 5 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:58:06.061215 8148 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:58:06.060868 8139 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: configured and running, proceeding with master startup.
I20250110 20:58:06.158200 8015 consensus_queue.cc:1035] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:58:06.164567 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
I20250110 20:58:06.195410 7911 raft_consensus.cc:491] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:06.195883 7911 raft_consensus.cc:513] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:06.197772 7911 leader_election.cc:290] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:06.199802 7208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ccff4e5d84861b6dc6a6cb5142416" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8" is_pre_election: true
I20250110 20:58:06.200887 7208 raft_consensus.cc:2463] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:06.202488 7435 leader_election.cc:304] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:06.203374 8039 raft_consensus.cc:2798] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:06.203347 8017 consensus_queue.cc:1035] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:06.203697 8039 raft_consensus.cc:491] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:06.204002 8039 raft_consensus.cc:3054] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:06.206876 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ccff4e5d84861b6dc6a6cb5142416" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:06.207439 7353 raft_consensus.cc:2463] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 0.
I20250110 20:58:06.236503 8015 consensus_queue.cc:1035] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:06.241734 8039 raft_consensus.cc:513] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:06.247360 7207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ccff4e5d84861b6dc6a6cb5142416" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "244d9ba396f549f2a5015792b6c5fff8"
I20250110 20:58:06.247929 7207 raft_consensus.cc:3054] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:06.256358 7207 raft_consensus.cc:2463] T f08ccff4e5d84861b6dc6a6cb5142416 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:06.258016 7435 leader_election.cc:304] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:06.261585 7911 raft_consensus.cc:2798] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:06.268481 7353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08ccff4e5d84861b6dc6a6cb5142416" candidate_uuid: "c3b63d95ca734d8a9803b96bfd94f307" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:06.268986 7353 raft_consensus.cc:3054] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:06.270370 8039 leader_election.cc:290] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 election: Requested vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:06.274015 8087 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" instance_seqno: 1736542670266110) as {username='slave'} at 127.4.171.194:46653; Asking this server to re-register.
I20250110 20:58:06.276196 7353 raft_consensus.cc:2463] T f08ccff4e5d84861b6dc6a6cb5142416 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b63d95ca734d8a9803b96bfd94f307 in term 1.
I20250110 20:58:06.279783 7399 heartbeater.cc:463] Registering TS with master...
I20250110 20:58:06.281803 7911 raft_consensus.cc:695] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 LEADER]: Becoming Leader. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Running, Role: LEADER
I20250110 20:58:06.282631 7911 consensus_queue.cc:237] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:06.293300 8017 consensus_queue.cc:1035] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:06.327579 8087 ts_manager.cc:194] Registered new tserver with Master: 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035)
I20250110 20:58:06.342716 8087 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" instance_seqno: 1736542672062569) as {username='slave'} at 127.4.171.195:41725; Asking this server to re-register.
I20250110 20:58:06.357064 7547 heartbeater.cc:463] Registering TS with master...
I20250110 20:58:06.360360 8087 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" instance_seqno: 1736542668472492) as {username='slave'} at 127.4.171.193:49803; Asking this server to re-register.
I20250110 20:58:06.363032 8086 ts_manager.cc:194] Registered new tserver with Master: c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:06.371948 7256 heartbeater.cc:463] Registering TS with master...
I20250110 20:58:06.377424 8087 ts_manager.cc:194] Registered new tserver with Master: 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
I20250110 20:58:06.438156 8165 catalog_manager.cc:1260] Loaded cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:58:06.438570 8165 catalog_manager.cc:1553] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: loading cluster ID for follower catalog manager: success
I20250110 20:58:06.452478 8148 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:06.453011 8148 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:06.455883 8148 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:06.459405 8165 catalog_manager.cc:1575] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: acquiring CA information for follower catalog manager: success
I20250110 20:58:06.482949 8165 catalog_manager.cc:1603] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250110 20:58:06.487951 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:06.488930 6851 raft_consensus.cc:2405] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f9bb61bdf30044a1b482d6a66e29435f for term 6 because replica has last-logged OpId of term: 5 index: 185, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20250110 20:58:06.612012 6828 leader_election.cc:336] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: RequestConsensusVote RPC to 127.4.171.252:33621 timed out after 0.505s (SENT)
I20250110 20:58:06.612447 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51; no voters: a36f292a1a20461e9aef5aefe8fd0f08, f9bb61bdf30044a1b482d6a66e29435f
I20250110 20:58:06.630410 8176 raft_consensus.cc:2743] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20250110 20:58:07.143081 8028 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.686s (ON_OUTBOUND_QUEUE)
I20250110 20:58:07.149917 8178 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:58:07.150538 7926 consensus_queue.cc:1035] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:07.150604 8178 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:07.152364 8178 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:07.188336 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 6 candidate_status { last_received { term: 5 index: 185 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f" is_pre_election: true
I20250110 20:58:07.189173 8103 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5496a4fb46f84c47a170dfe492f14b51 in term 5.
W20250110 20:58:07.189527 7251 debug-util.cc:398] Leaking SignalData structure 0x7b0800120180 after lost signal to thread 7107
W20250110 20:58:07.190582 7251 debug-util.cc:398] Leaking SignalData structure 0x7b080011dba0 after lost signal to thread 7254
I20250110 20:58:07.193312 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, f9bb61bdf30044a1b482d6a66e29435f; no voters:
I20250110 20:58:07.194067 8178 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20250110 20:58:07.194378 8178 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Starting leader election (detected failure of leader a36f292a1a20461e9aef5aefe8fd0f08)
I20250110 20:58:07.194684 8178 raft_consensus.cc:3054] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 5 FOLLOWER]: Advancing to term 6
I20250110 20:58:07.195911 7911 consensus_queue.cc:1035] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250110 20:58:07.220216 7926 consensus_queue.cc:1035] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P 9ef4ce3d44cc486fb0ee0ad85127011b [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:58:07.233788 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
I20250110 20:58:07.237581 8178 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:07.243162 8178 leader_election.cc:290] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 election: Requested vote from peers f9bb61bdf30044a1b482d6a66e29435f (127.4.171.253:40203), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:07.244007 8103 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5496a4fb46f84c47a170dfe492f14b51" candidate_term: 6 candidate_status { last_received { term: 5 index: 185 } } ignore_live_leader: false dest_uuid: "f9bb61bdf30044a1b482d6a66e29435f"
I20250110 20:58:07.244580 8103 raft_consensus.cc:3054] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 5 FOLLOWER]: Advancing to term 6
I20250110 20:58:07.269304 8171 consensus_queue.cc:1035] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 [LEADER]: Connected to new peer: Peer: permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250110 20:58:07.274116 8103 raft_consensus.cc:2463] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5496a4fb46f84c47a170dfe492f14b51 in term 6.
I20250110 20:58:07.283882 6828 leader_election.cc:304] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5496a4fb46f84c47a170dfe492f14b51, f9bb61bdf30044a1b482d6a66e29435f; no voters:
I20250110 20:58:07.292300 8178 raft_consensus.cc:2798] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 6 FOLLOWER]: Leader election won for term 6
I20250110 20:58:07.303855 8178 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [term 6 LEADER]: Becoming Leader. State: Replica: 5496a4fb46f84c47a170dfe492f14b51, State: Running, Role: LEADER
I20250110 20:58:07.304661 8178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 185, Committed index: 185, Last appended: 5.185, Last appended by leader: 185, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:07.338491 8190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5496a4fb46f84c47a170dfe492f14b51. Latest consensus state: current_term: 6 leader_uuid: "5496a4fb46f84c47a170dfe492f14b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:58:07.339337 8190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: LEADER
I20250110 20:58:07.347486 8193 catalog_manager.cc:1476] Loading table and tablet metadata into memory...
I20250110 20:58:07.366322 8193 catalog_manager.cc:670] Loaded metadata for table default.table_e760ae0763c04467a8fea38edff37362 [id=150bdbba49cc4d7b80896013ce4acadd]
I20250110 20:58:07.368711 8193 catalog_manager.cc:670] Loaded metadata for table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]
I20250110 20:58:07.370343 8193 catalog_manager.cc:670] Loaded metadata for table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401]
I20250110 20:58:07.371842 8193 catalog_manager.cc:670] Loaded metadata for table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb]
I20250110 20:58:07.373520 8193 catalog_manager.cc:670] Loaded metadata for table default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da]
I20250110 20:58:07.375047 8193 catalog_manager.cc:670] Loaded metadata for table default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75]
I20250110 20:58:07.376744 8193 catalog_manager.cc:670] Loaded metadata for table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]
I20250110 20:58:07.379024 8193 catalog_manager.cc:670] Loaded metadata for table default.table_eed9317ae3914199a2cdc0d7e112371f [id=c937038ba33645559692b4937a1fd7b6]
I20250110 20:58:07.380618 8193 catalog_manager.cc:670] Loaded metadata for table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047]
I20250110 20:58:07.386389 8193 catalog_manager.cc:670] Loaded metadata for table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e]
I20250110 20:58:07.388020 8193 catalog_manager.cc:670] Loaded metadata for table default.table_34c642de1a56476a83c2b97f9cdd1287 [id=f089167c681940c1a2f841509f560366]
I20250110 20:58:07.390348 8193 catalog_manager.cc:670] Loaded metadata for table default.table_4dac0491728e4031866a732faeb6d42c [id=f974e7b92b494a63a9544aaad6c79f07]
I20250110 20:58:07.428720 8015 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:07.429214 8015 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:07.431129 8015 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:07.435281 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:58:07.436430 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:07.436764 7287 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:07.437831 7926 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:07.442739 8193 tablet_loader.cc:96] loaded metadata for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_59954cb851e943ff9f10bca64b86d8b8 [id=6ccc906dcedf49caad8b9efe99546401])
I20250110 20:58:07.449023 8193 tablet_loader.cc:96] loaded metadata for tablet 47accefc003b4e2eae08990302350167 (table default.table_e760ae0763c04467a8fea38edff37362 [id=150bdbba49cc4d7b80896013ce4acadd])
I20250110 20:58:07.452406 8193 tablet_loader.cc:96] loaded metadata for tablet 49d26618d4ee4608a9a0333929b654f4 (table default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b])
I20250110 20:58:07.458300 8193 tablet_loader.cc:96] loaded metadata for tablet 533494271bfa4d7486026a00767cf879 (table default.table_df681f6d6d9e483282a9b28e42817aef [id=e3071b63c99e4526910dae97667de29e])
I20250110 20:58:07.459628 8193 tablet_loader.cc:96] loaded metadata for tablet 721c67636cbc4fe98742a00b970f3701 (table default.table_34c642de1a56476a83c2b97f9cdd1287 [id=f089167c681940c1a2f841509f560366])
I20250110 20:58:07.464910 8193 tablet_loader.cc:96] loaded metadata for tablet 8c518fd5cc6a4d1e8f7fc47489fe42a8 (table default.table_dca3421fe79d4330bc146c208b21a6f2 [id=bbced07fb45947d4aaa1e1e4a08756da])
I20250110 20:58:07.466970 8193 tablet_loader.cc:96] loaded metadata for tablet 9501a5e3c7354c45b8b09ef28e294dc1 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376])
I20250110 20:58:07.468207 8193 tablet_loader.cc:96] loaded metadata for tablet ab2f24f374cb4dfba6d9563cb6666f6c (table default.table_4dac0491728e4031866a732faeb6d42c [id=f974e7b92b494a63a9544aaad6c79f07])
I20250110 20:58:07.469691 8193 tablet_loader.cc:96] loaded metadata for tablet ac7a9bac7d9c434e9a8b3b9d5b67f4a5 (table default.table_eed9317ae3914199a2cdc0d7e112371f [id=c937038ba33645559692b4937a1fd7b6])
I20250110 20:58:07.470898 8193 tablet_loader.cc:96] loaded metadata for tablet f08ccff4e5d84861b6dc6a6cb5142416 (table default.table_cee5a6c92d284bf2ab6f7eb541bbb600 [id=c21c62e71511487db217a6f65b6abb75])
I20250110 20:58:07.472162 8193 tablet_loader.cc:96] loaded metadata for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047])
I20250110 20:58:07.474472 8193 tablet_loader.cc:96] loaded metadata for tablet fb8d5bf6a9694240b10e1b1c178b528b (table default.table_1073e5a3e4b84cbc9198aa13f15af95e [id=b0a900b933d349bc8b9b5fc9ce31decb])
I20250110 20:58:07.478783 8193 catalog_manager.cc:1485] Initializing Kudu cluster ID...
I20250110 20:58:07.495704 8193 catalog_manager.cc:1260] Loaded cluster ID: 5a7dc727eb864088baf8c48362a06057
I20250110 20:58:07.496042 8193 catalog_manager.cc:1496] Initializing Kudu internal certificate authority...
I20250110 20:58:07.513836 8193 catalog_manager.cc:1505] Loading token signing keys...
I20250110 20:58:07.535568 8193 catalog_manager.cc:5910] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: Loaded TSK: 0
I20250110 20:58:07.536616 8193 catalog_manager.cc:1515] Initializing in-progress tserver states...
I20250110 20:58:07.550201 8193 catalog_manager.cc:1528] Loading latest processed Hive Metastore notification log event ID...
I20250110 20:58:07.565980 8193 catalog_manager.cc:5748] Last processed Hive Metastore notification event ID: 23
I20250110 20:58:07.615530 8103 raft_consensus.cc:1270] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Refusing update from remote peer 5496a4fb46f84c47a170dfe492f14b51: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 187. (index mismatch)
I20250110 20:58:07.617224 8190 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 186, Last known committed idx: 6, Time since last communication: 0.000s
W20250110 20:58:07.804136 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:07.859138 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.686s (ON_OUTBOUND_QUEUE)
I20250110 20:58:07.859575 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:07.989993 8197 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20250110 20:58:07.998064 8195 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5496a4fb46f84c47a170dfe492f14b51. Latest consensus state: current_term: 6 leader_uuid: "5496a4fb46f84c47a170dfe492f14b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:58:08.001291 8195 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:58:08.050360 8195 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 6 leader_uuid: "5496a4fb46f84c47a170dfe492f14b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:58:08.051069 8195 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [sys.catalog]: This master's current role is: FOLLOWER
I20250110 20:58:08.080215 8190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 6 leader_uuid: "5496a4fb46f84c47a170dfe492f14b51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } } }
I20250110 20:58:08.080953 8190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [sys.catalog]: This master's current role is: LEADER
W20250110 20:58:08.174759 6828 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.171.252:33621, user_credentials={real_user=slave}} blocked reactor thread for 114226us
I20250110 20:58:08.195190 7188 tablet_service.cc:1467] Processing CreateTablet for tablet 49d26618d4ee4608a9a0333929b654f4 (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:08.196561 7188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d26618d4ee4608a9a0333929b654f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:08.252584 7926 raft_consensus.cc:491] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:08.253060 7926 raft_consensus.cc:513] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.255504 7926 leader_election.cc:290] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:08.257890 7287 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
W20250110 20:58:08.253687 7068 catalog_manager.cc:5786] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
I20250110 20:58:08.298400 7548 heartbeater.cc:502] Master 127.4.171.254:36415 was elected leader, sending a full tablet report...
W20250110 20:58:08.316043 7288 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:08.316439 7288 leader_election.cc:304] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:08.317577 7926 raft_consensus.cc:2743] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:08.320477 6828 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_58cf4fbf7d1a441ebe2a9611029cf17b [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
W20250110 20:58:08.326155 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:08.326468 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:08.326699 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:08.346115 7400 heartbeater.cc:502] Master 127.4.171.254:36415 was elected leader, sending a full tablet report...
I20250110 20:58:08.360531 7260 heartbeater.cc:502] Master 127.4.171.254:36415 was elected leader, sending a full tablet report...
W20250110 20:58:08.370784 6841 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:08.371352 6841 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:08.332218 6842 catalog_manager.cc:5526] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } }
I20250110 20:58:08.376037 8212 tablet_bootstrap.cc:492] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8: Bootstrap starting.
I20250110 20:58:08.388111 7332 tablet_service.cc:1467] Processing CreateTablet for tablet 49d26618d4ee4608a9a0333929b654f4 (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:08.389019 8212 tablet_bootstrap.cc:654] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:08.389092 7332 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d26618d4ee4608a9a0333929b654f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:08.407964 8212 tablet_bootstrap.cc:492] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8: No bootstrap required, opened a new log
I20250110 20:58:08.408542 8212 ts_tablet_manager.cc:1397] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.000s
I20250110 20:58:08.408612 4783 master-stress-test.cc:577] Killing test thread 1/10
I20250110 20:58:08.411242 8212 raft_consensus.cc:357] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.416683 8212 raft_consensus.cc:383] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:08.417025 8212 raft_consensus.cc:738] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Initialized, Role: FOLLOWER
W20250110 20:58:08.386783 6840 catalog_manager.cc:5359] Ignoring report from unknown tablet 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:08.417786 8212 consensus_queue.cc:260] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:58:08.418398 6840 catalog_manager.cc:5359] Ignoring report from unknown tablet 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:08.383149 6842 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_58cf4fbf7d1a441ebe2a9611029cf17b [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 2 got 0
I20250110 20:58:08.420358 8212 ts_tablet_manager.cc:1428] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20250110 20:58:08.420279 6842 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_c3ed0987ec6e41d39ddc47cba082a635 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 2 got 1
I20250110 20:58:08.421172 6842 catalog_manager.cc:5526] T ab2f24f374cb4dfba6d9563cb6666f6c P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } }
I20250110 20:58:08.422580 6842 catalog_manager.cc:5526] T ac7a9bac7d9c434e9a8b3b9d5b67f4a5 P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } }
I20250110 20:58:08.429023 7481 tablet_service.cc:1467] Processing CreateTablet for tablet 49d26618d4ee4608a9a0333929b654f4 (DEFAULT_TABLE table=default.table_35a55b6fae704e909310ca9089810b45 [id=666e7a40ae294813a1bba6474602377b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250110 20:58:08.430017 7481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d26618d4ee4608a9a0333929b654f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250110 20:58:08.458652 6842 catalog_manager.cc:5526] T f08ccff4e5d84861b6dc6a6cb5142416 P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195). New cstate: current_term: 1 leader_uuid: "c3b63d95ca734d8a9803b96bfd94f307" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: HEALTHY } } }
I20250110 20:58:08.466238 6842 catalog_manager.cc:5526] T 8c518fd5cc6a4d1e8f7fc47489fe42a8 P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } }
I20250110 20:58:08.474076 6842 catalog_manager.cc:5526] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194). New cstate: current_term: 1 leader_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } }
I20250110 20:58:08.480581 6842 catalog_manager.cc:5618] TS c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717) does not have the latest schema for tablet 9501a5e3c7354c45b8b09ef28e294dc1 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:08.489358 8215 tablet_bootstrap.cc:492] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b: Bootstrap starting.
I20250110 20:58:08.492076 8216 tablet_bootstrap.cc:492] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307: Bootstrap starting.
I20250110 20:58:08.546119 8216 tablet_bootstrap.cc:654] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:08.549765 8215 tablet_bootstrap.cc:654] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b: Neither blocks nor log segments found. Creating new log.
I20250110 20:58:08.565553 6841 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_58cf4fbf7d1a441ebe2a9611029cf17b [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 2 got 0
I20250110 20:58:08.566852 6841 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_c3ed0987ec6e41d39ddc47cba082a635 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 2 got 1
I20250110 20:58:08.567693 6841 catalog_manager.cc:5618] TS 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745) does not have the latest schema for tablet 9501a5e3c7354c45b8b09ef28e294dc1 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:08.578287 6840 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_58cf4fbf7d1a441ebe2a9611029cf17b [id=ca7388d37c1f4aa8ac6ff138c21c7047]). Expected version 2 got 0
I20250110 20:58:08.579397 6840 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 1c82412f7ba946b9b0dcffd8fa883353 (table default.table_c3ed0987ec6e41d39ddc47cba082a635 [id=6ccc906dcedf49caad8b9efe99546401]). Expected version 2 got 1
I20250110 20:58:08.580731 6840 catalog_manager.cc:5618] TS 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035) does not have the latest schema for tablet 9501a5e3c7354c45b8b09ef28e294dc1 (table default.table_176fb71364f24095a1efbf0d29c18ff8 [id=c4582c5662964616b35896bcce266376]). Expected version 1 got 0
I20250110 20:58:08.602053 8215 tablet_bootstrap.cc:492] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b: No bootstrap required, opened a new log
I20250110 20:58:08.602885 8215 ts_tablet_manager.cc:1397] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent bootstrapping tablet: real 0.118s user 0.011s sys 0.003s
I20250110 20:58:08.610595 8216 tablet_bootstrap.cc:492] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307: No bootstrap required, opened a new log
I20250110 20:58:08.611071 8216 ts_tablet_manager.cc:1397] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307: Time spent bootstrapping tablet: real 0.121s user 0.006s sys 0.008s
I20250110 20:58:08.613602 8216 raft_consensus.cc:357] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.626549 8215 raft_consensus.cc:357] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.633457 8216 raft_consensus.cc:383] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:08.637552 8216 raft_consensus.cc:738] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b63d95ca734d8a9803b96bfd94f307, State: Initialized, Role: FOLLOWER
I20250110 20:58:08.641728 8215 raft_consensus.cc:383] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250110 20:58:08.643481 8215 raft_consensus.cc:738] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ef4ce3d44cc486fb0ee0ad85127011b, State: Initialized, Role: FOLLOWER
I20250110 20:58:08.642930 8216 consensus_queue.cc:260] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.650760 8147 raft_consensus.cc:491] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:08.651217 8147 raft_consensus.cc:513] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.654888 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d26618d4ee4608a9a0333929b654f4" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" is_pre_election: true
I20250110 20:58:08.655157 8147 leader_election.cc:290] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:08.655774 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d26618d4ee4608a9a0333929b654f4" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307" is_pre_election: true
I20250110 20:58:08.656333 7501 raft_consensus.cc:2463] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:08.659197 7145 leader_election.cc:304] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307; no voters:
I20250110 20:58:08.659878 8147 raft_consensus.cc:2798] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250110 20:58:08.660182 8147 raft_consensus.cc:491] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250110 20:58:08.660485 8147 raft_consensus.cc:3054] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:08.668256 8147 raft_consensus.cc:513] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.647459 8215 consensus_queue.cc:260] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.673692 7351 raft_consensus.cc:2463] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 0.
I20250110 20:58:08.681692 8216 ts_tablet_manager.cc:1428] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307: Time spent starting tablet: real 0.070s user 0.008s sys 0.000s
I20250110 20:58:08.691913 8215 ts_tablet_manager.cc:1428] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b: Time spent starting tablet: real 0.086s user 0.007s sys 0.000s
I20250110 20:58:08.696605 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d26618d4ee4608a9a0333929b654f4" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b"
I20250110 20:58:08.697132 7351 raft_consensus.cc:3054] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:08.700582 7501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49d26618d4ee4608a9a0333929b654f4" candidate_uuid: "244d9ba396f549f2a5015792b6c5fff8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b63d95ca734d8a9803b96bfd94f307"
I20250110 20:58:08.701077 7501 raft_consensus.cc:3054] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Advancing to term 1
I20250110 20:58:08.700474 8147 leader_election.cc:290] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Requested vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
I20250110 20:58:08.729461 7351 raft_consensus.cc:2463] T 49d26618d4ee4608a9a0333929b654f4 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:08.730654 7145 leader_election.cc:304] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b; no voters:
I20250110 20:58:08.731410 8147 raft_consensus.cc:2798] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 FOLLOWER]: Leader election won for term 1
I20250110 20:58:08.734421 7501 raft_consensus.cc:2463] T 49d26618d4ee4608a9a0333929b654f4 P c3b63d95ca734d8a9803b96bfd94f307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 244d9ba396f549f2a5015792b6c5fff8 in term 1.
I20250110 20:58:08.844210 8225 tablet.cc:1721] T 1c82412f7ba946b9b0dcffd8fa883353 P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:58:08.846621 8226 tablet.cc:1721] T 1c82412f7ba946b9b0dcffd8fa883353 P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:58:08.851603 8232 tablet.cc:1721] T 1c82412f7ba946b9b0dcffd8fa883353 P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:58:08.861128 8234 tablet.cc:1721] T 9501a5e3c7354c45b8b09ef28e294dc1 P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:08.870841 8236 tablet.cc:1721] T 9501a5e3c7354c45b8b09ef28e294dc1 P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:08.894721 8147 raft_consensus.cc:695] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [term 1 LEADER]: Becoming Leader. State: Replica: 244d9ba396f549f2a5015792b6c5fff8, State: Running, Role: LEADER
I20250110 20:58:08.896162 8147 consensus_queue.cc:237] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:08.899667 8234 tablet.cc:1721] T f58d4b7a1ff0439a9b1a08e9e04c044d P 9ef4ce3d44cc486fb0ee0ad85127011b: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250110 20:58:08.908056 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:08.913507 8240 tablet.cc:1721] T f58d4b7a1ff0439a9b1a08e9e04c044d P c3b63d95ca734d8a9803b96bfd94f307: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:58:08.955036 6839 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: default.table_58cf4fbf7d1a441ebe2a9611029cf17b [id=ca7388d37c1f4aa8ac6ff138c21c7047] alter complete (version 2)
I20250110 20:58:08.978248 8241 tablet.cc:1721] T f58d4b7a1ff0439a9b1a08e9e04c044d P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250110 20:58:08.992925 8242 tablet.cc:1721] T 9501a5e3c7354c45b8b09ef28e294dc1 P 244d9ba396f549f2a5015792b6c5fff8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250110 20:58:09.024025 6842 catalog_manager.cc:5526] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193). New cstate: current_term: 1 leader_uuid: "244d9ba396f549f2a5015792b6c5fff8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } health_report { overall_health: UNKNOWN } } }
W20250110 20:58:09.047479 6828 leader_election.cc:336] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: RequestConsensusVote RPC to 127.4.171.252:33621 timed out after 0.817s (SENT)
I20250110 20:58:09.100477 6839 catalog_manager.cc:6071] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51: default.table_c3ed0987ec6e41d39ddc47cba082a635 [id=6ccc906dcedf49caad8b9efe99546401] alter complete (version 2)
W20250110 20:58:09.194980 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:09.322167 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
W20250110 20:58:09.341099 6828 leader_election.cc:336] T 00000000000000000000000000000000 P 5496a4fb46f84c47a170dfe492f14b51 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: RequestConsensusVote RPC to 127.4.171.252:33621 timed out after 1.056s (SENT)
I20250110 20:58:09.553431 8196 mvcc.cc:204] Tried to move back new op lower bound from 7112878848455778304 to 7112878847208153088. Current Snapshot: MvccSnapshot[applied={T|T < 7112878848455778304 or (T in {7112878848455778304})}]
W20250110 20:58:09.606866 8167 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:58:06.456306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.252:33621 (local address 127.0.0.1:59034)
0110 20:58:06.483599 (+ 27293us) negotiation.cc:107] Waiting for socket to connect
0110 20:58:06.483645 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0110 20:58:06.483896 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:58:09.605786 (+3121890us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":9289,"thread_start_us":28125,"threads_started":1}
I20250110 20:58:09.610006 8250 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:09.610441 8250 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:09.612210 8250 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:09.630385 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:09.611985 7607 master_runner.cc:186] Error getting master registration for 127.4.171.252:33621: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes,
I20250110 20:58:09.695958 7607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250110 20:58:09.760044 8147 consensus_queue.cc:1035] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:58:09.767405 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:09.775861 6873 debug-util.cc:398] Leaking SignalData structure 0x7b08000731e0 after lost signal to thread 6810
W20250110 20:58:09.776526 6873 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4480 after lost signal to thread 6876
I20250110 20:58:09.820720 8147 consensus_queue.cc:1035] T 49d26618d4ee4608a9a0333929b654f4 P 244d9ba396f549f2a5015792b6c5fff8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250110 20:58:10.302804 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:10.547648 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:10.821369 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.594s (ON_OUTBOUND_QUEUE)
I20250110 20:58:10.821748 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:10.823019 8270 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:11.069964 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:11.192934 8271 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:11.193673 8271 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:11.196918 8271 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:11.198007 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:11.198576 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:11.198866 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:11.199816 8271 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:11.341277 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:12.051268 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:12.297044 8277 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:12.297523 8277 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:12.299837 8277 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:12.304301 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
W20250110 20:58:12.308909 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:12.682225 8167 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:58:09.639501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.252:33621 (local address 127.0.0.1:59036)
0110 20:58:09.640636 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0110 20:58:09.640677 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0110 20:58:09.640885 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:58:12.681740 (+3040855us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":443}
W20250110 20:58:12.682696 8028 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes [suppressed 1 similar messages]
W20250110 20:58:13.305536 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:13.386982 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.701s (ON_OUTBOUND_QUEUE)
I20250110 20:58:13.387424 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:13.390471 8286 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:13.463935 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:13.464401 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
I20250110 20:58:13.531312 8282 raft_consensus.cc:491] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:13.531912 8282 raft_consensus.cc:513] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:13.533776 8282 leader_election.cc:290] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:13.535843 7287 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
W20250110 20:58:13.536396 7288 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:13.536736 7288 leader_election.cc:304] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:13.537250 8282 raft_consensus.cc:2743] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:13.766580 8282 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:13.766988 8282 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:13.768394 8282 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:13.769642 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:58:13.770113 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:13.770367 7287 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:13.770824 8282 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:14.324180 7436 consensus_peers.cc:487] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a. This is attempt 11: this message will repeat every 5th retry.
W20250110 20:58:14.331810 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:14.381193 8291 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:14.382001 8291 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:14.384847 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:14.385543 8291 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
W20250110 20:58:14.498618 7436 consensus_peers.cc:487] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5. This is attempt 11: this message will repeat every 5th retry.
W20250110 20:58:14.501688 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:15.395221 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:15.414307 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:15.710315 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.661s (ON_OUTBOUND_QUEUE)
I20250110 20:58:15.710795 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:15.712188 8299 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:16.389025 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:16.511056 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:17.001982 8305 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:17.002447 8305 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:17.003971 8305 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:17.004614 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
W20250110 20:58:17.145607 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:17.286866 8167 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:58:12.684634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.252:33621 (local address 127.0.0.1:45990)
0110 20:58:12.685288 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0110 20:58:12.685328 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0110 20:58:12.685564 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:58:17.285991 (+4600427us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":79}
I20250110 20:58:17.288655 7607 master_runner.cc:186] Error getting master registration for 127.4.171.252:33621: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes,
I20250110 20:58:17.289100 7607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250110 20:58:17.502875 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:17.969856 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.680s (ON_OUTBOUND_QUEUE)
I20250110 20:58:17.970294 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:17.971356 8312 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:18.112362 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:18.387501 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:19.257020 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:19.265695 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:19.365813 8320 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:19.366230 8320 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:19.368013 8320 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:19.368695 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:19.455885 8318 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:19.456254 8318 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:19.457716 8318 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:19.459937 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:19.460378 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:19.460660 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:19.461127 8318 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:20.105813 8028 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.736s (ON_OUTBOUND_QUEUE) [suppressed 2 similar messages]
W20250110 20:58:20.146512 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:20.246932 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:20.358587 8167 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:58:17.289556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.252:33621 (local address 127.0.0.1:46002)
0110 20:58:17.290262 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0110 20:58:17.290292 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0110 20:58:17.290522 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:58:20.357827 (+3067305us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":86}
W20250110 20:58:20.359228 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes
I20250110 20:58:20.359684 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:20.361575 8329 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:21.078425 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:21.352531 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:21.677377 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
W20250110 20:58:22.120784 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:22.225951 8337 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:22.226338 8337 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:22.228345 8337 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:22.229354 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
W20250110 20:58:22.593173 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:23.385619 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:23.413684 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.587s (ON_OUTBOUND_QUEUE)
I20250110 20:58:23.414113 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:23.428304 8346 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:23.649989 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:24.363859 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:24.364694 7436 consensus_peers.cc:487] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a. This is attempt 16: this message will repeat every 5th retry.
W20250110 20:58:24.611001 7436 consensus_peers.cc:487] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5. This is attempt 16: this message will repeat every 5th retry.
W20250110 20:58:24.616477 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:25.234453 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:25.546094 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:25.596087 8282 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:25.596647 8282 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:25.599071 8282 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:25.600441 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:58:25.600972 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:25.601253 7287 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:25.601744 8282 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:25.991256 8356 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:25.991950 8356 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:25.994053 8356 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:25.995005 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
W20250110 20:58:26.045692 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:26.422859 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:26.520236 8028 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.525s (ON_OUTBOUND_QUEUE) [suppressed 1 similar messages]
W20250110 20:58:26.807983 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:27.051033 8028 leader_election.cc:336] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621): Timed out: connection negotiation to 127.4.171.252:33621 for RPC RequestConsensusVote timed out after 0.525s (ON_OUTBOUND_QUEUE)
I20250110 20:58:27.051507 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:27.052762 8365 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:27.274550 8338 negotiation.cc:337] Failed RPC negotiation. Trace:
0110 20:58:22.228769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.171.252:33621 (local address 127.0.0.1:46200)
0110 20:58:22.235864 (+ 7095us) negotiation.cc:107] Waiting for socket to connect
0110 20:58:22.235901 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0110 20:58:22.236088 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0110 20:58:27.273846 (+5037758us) negotiation.cc:327] Negotiation complete: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes
Metrics: {"client-negotiator.queue_time_us":6535,"thread_start_us":6314,"threads_started":1}
W20250110 20:58:27.274847 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:27.275224 7607 master_runner.cc:186] Error getting master registration for 127.4.171.252:33621: Timed out: Client connection negotiation failed: client connection to 127.4.171.252:33621: received 0 of 4 requested bytes,
I20250110 20:58:27.275780 7607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250110 20:58:27.839118 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:27.987890 8361 raft_consensus.cc:491] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:27.988272 8361 raft_consensus.cc:513] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:27.989711 8361 leader_election.cc:290] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:27.991009 7287 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
W20250110 20:58:27.991431 7288 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:27.991711 7288 leader_election.cc:304] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:27.992301 8361 raft_consensus.cc:2743] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:28.071719 8369 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:28.072144 8369 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:28.073942 8369 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:28.074525 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:28.085357 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:28.086719 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:28.087317 8369 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:28.394385 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:29.014676 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:29.356668 8377 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:29.357242 8377 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
W20250110 20:58:29.362840 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:29.363559 8377 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:29.363706 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:29.364085 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:29.364742 8377 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:29.393710 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:29.534417 8380 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:29.534896 8380 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:29.536790 8380 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:29.537768 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:29.539927 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:29.544085 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:29.545094 8380 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:58:30.048991 8384 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:30.049521 8384 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:30.051339 8384 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:30.052460 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:30.052693 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:30.054212 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:30.054872 8384 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:30.116647 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:30.171990 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:31.048640 8390 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:31.049043 8390 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:31.050475 8390 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:31.051191 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:31.051676 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:31.052873 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:31.053457 8390 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:31.218029 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:31.249923 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:32.099978 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:32.104607 8375 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:32.105064 8375 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:32.106606 8375 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:32.107929 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:58:32.108408 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:32.108752 7287 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:32.109373 8375 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:32.261945 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:32.561492 8398 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:32.561961 8398 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:32.563858 8398 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:32.564747 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:32.564766 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:32.566215 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:32.566900 8398 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:33.180032 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:33.211840 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:34.012241 8405 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:34.012653 8405 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:34.014446 8405 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:34.015134 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:34.015621 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:34.017390 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:34.018111 8405 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:34.105222 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:34.105916 7436 consensus_peers.cc:487] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a. This is attempt 21: this message will repeat every 5th retry.
W20250110 20:58:34.322149 7436 consensus_peers.cc:487] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5. This is attempt 21: this message will repeat every 5th retry.
W20250110 20:58:34.324162 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:34.980657 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:35.027130 8415 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:35.027741 8415 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:35.029640 8415 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:35.030323 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:35.030691 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:35.032274 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:35.032886 8415 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:35.283905 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:35.561945 8417 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:35.562477 8417 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:35.564448 8417 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:35.565374 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:35.565579 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:35.566977 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:35.568364 8417 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:36.042752 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:36.215327 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:36.636674 8422 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:36.637137 8422 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:36.640161 8422 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:36.640964 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:36.641499 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:36.642671 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:36.643323 8422 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:36.971947 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:37.401751 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:37.614795 6828 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: UpdateConsensus RPC to 127.4.171.252:33621 timed out after 30.000s (SENT) [suppressed 3 similar messages]
W20250110 20:58:37.826498 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:38.096745 6978 catalog_manager.cc:4670] Async tablet task AlterTable RPC for tablet f58d4b7a1ff0439a9b1a08e9e04c044d (table default.table_ba5d6def0428488589bda04057a79fab [id=ca7388d37c1f4aa8ac6ff138c21c7047], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f58d4b7a1ff0439a9b1a08e9e04c044d consensus state has no leader
W20250110 20:58:38.457793 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:38.646564 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:39.002892 8420 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:39.003371 8420 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:39.004951 8420 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:39.006439 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:39.006939 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:39.007220 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:39.007717 8420 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:39.335772 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:39.853509 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:40.284057 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:40.738090 8440 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:40.738550 8440 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:40.741006 8440 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:40.742019 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:40.742085 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:40.743631 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:40.744302 8440 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:40.847055 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:41.300177 8446 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:41.300696 8446 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:41.303440 8446 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:41.304541 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:41.306797 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
W20250110 20:58:41.307910 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:41.307951 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:41.309087 8446 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:41.860311 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:41.904647 8450 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:41.905122 8450 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:41.906697 8450 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:41.907496 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:41.907703 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:41.909143 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:41.909672 8450 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:42.360682 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:42.506084 8452 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:42.506556 8452 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:42.508345 8452 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:42.509022 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:42.509296 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:42.510592 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:42.511171 8452 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20250110 20:58:42.672781 8438 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:42.673275 8438 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:42.675040 8438 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:42.676359 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:42.676808 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:42.677160 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:42.677680 8438 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:42.722679 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:43.036454 8455 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:43.036921 8455 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:43.038753 8455 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:43.039336 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:43.039664 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:43.040900 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:43.041435 8455 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:43.322997 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:43.790683 7394 debug-util.cc:398] Leaking SignalData structure 0x7b0800085d60 after lost signal to thread 7266
W20250110 20:58:43.791846 7394 debug-util.cc:398] Leaking SignalData structure 0x7b08000b6100 after lost signal to thread 7397
W20250110 20:58:43.911403 7436 consensus_peers.cc:487] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a. This is attempt 26: this message will repeat every 5th retry.
W20250110 20:58:43.913610 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:44.081597 8463 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:44.082053 8463 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:44.083693 8463 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:44.084385 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:44.085551 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:44.086990 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:44.087517 8463 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:44.111640 7436 consensus_peers.cc:487] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5. This is attempt 26: this message will repeat every 5th retry.
W20250110 20:58:44.114910 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:44.706825 8466 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:44.707415 8466 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:44.709949 8466 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:44.714248 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:44.715790 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:44.719837 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:44.720643 8466 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:45.018013 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:45.104028 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:45.790661 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:46.219890 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:46.799041 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:46.803988 8471 raft_consensus.cc:491] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:46.804397 8471 raft_consensus.cc:513] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:46.806744 8471 leader_election.cc:290] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:46.808046 7287 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
W20250110 20:58:46.808609 7288 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:46.808894 7288 leader_election.cc:304] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:46.809360 8471 raft_consensus.cc:2743] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:46.989389 8479 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:46.989905 8479 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:46.991796 8479 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:46.992512 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:46.992723 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:46.994213 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:46.994868 8479 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:47.103988 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:47.523871 8481 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:47.524324 8481 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:47.526640 8481 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:47.527774 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:47.528052 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:47.529321 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:47.529844 8481 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:47.747152 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:48.000737 8471 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:48.001186 8471 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:48.002993 8471 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:48.004494 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:58:48.004856 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:48.005219 7288 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:48.005859 8471 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:48.124742 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:48.420182 8473 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:48.420603 8473 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:48.422878 8473 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:48.427639 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:48.428521 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:48.428874 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:48.429487 8473 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250110 20:58:48.521070 8486 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:48.521548 8486 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:48.523373 8486 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:48.524044 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:48.524293 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:48.525574 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:48.526203 8486 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:48.670236 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:49.027941 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:49.518146 8492 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:49.518532 8492 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:49.521077 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:49.520110 8492 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:49.522176 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:49.523429 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:49.523916 8492 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:49.607245 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:50.014041 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:50.108289 8498 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:50.108773 8498 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:50.110436 8498 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:50.111405 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:50.111444 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:50.112829 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:50.113339 8498 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:50.570626 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:50.649089 8503 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:50.649585 8503 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:50.651295 8503 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:50.652633 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:50.652633 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:50.653967 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:50.654558 8503 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:50.995473 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:51.242013 8507 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:51.242486 8507 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:51.244318 8507 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:51.245280 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:51.245252 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:51.246665 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:51.247398 8507 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:51.606931 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:51.788425 8511 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:51.788898 8511 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:51.790551 8511 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:51.791219 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:51.791448 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:51.792590 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:51.793231 8511 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:51.949878 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:52.453454 7398 heartbeater.cc:643] Failed to heartbeat to 127.4.171.252:33621 (33 consecutive failures): Timed out: Failed to ping master at 127.4.171.252:33621: Ping RPC to 127.4.171.252:33621 timed out after 1.000s (SENT)
W20250110 20:58:52.815100 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:52.844069 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:52.967161 8471 raft_consensus.cc:491] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:52.967801 8471 raft_consensus.cc:513] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } }
I20250110 20:58:52.969619 8471 leader_election.cc:290] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:52.971277 7288 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
W20250110 20:58:52.971925 7287 leader_election.cc:343] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 02b0bf75f58e4a02a8c7def0bff5119f
I20250110 20:58:52.972251 7287 leader_election.cc:304] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:52.972911 8471 raft_consensus.cc:2743] T 02b0bf75f58e4a02a8c7def0bff5119f P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:53.151985 7546 heartbeater.cc:643] Failed to heartbeat to 127.4.171.252:33621 (47 consecutive failures): Timed out: Failed to ping master at 127.4.171.252:33621: Ping RPC to 127.4.171.252:33621 timed out after 1.000s (SENT)
I20250110 20:58:53.193735 8519 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:53.194226 8519 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:53.196012 8519 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:53.196525 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:53.196849 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:53.198043 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:53.198531 8519 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:53.621235 7436 consensus_peers.cc:487] T 69f0a5ceae3a489b9b0552c550d3e80a P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69f0a5ceae3a489b9b0552c550d3e80a. This is attempt 31: this message will repeat every 5th retry.
W20250110 20:58:53.624936 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:53.796818 8524 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:53.797277 8524 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:53.799166 8524 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:53.799786 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:53.800537 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:53.801939 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:53.802448 8524 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:53.819823 7436 consensus_peers.cc:487] T 72de597ddab241fbbb82fc04e40385c5 P c3b63d95ca734d8a9803b96bfd94f307 -> Peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Couldn't send request to peer 9ef4ce3d44cc486fb0ee0ad85127011b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72de597ddab241fbbb82fc04e40385c5. This is attempt 31: this message will repeat every 5th retry.
W20250110 20:58:53.822707 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:58:53.892128 7255 heartbeater.cc:643] Failed to heartbeat to 127.4.171.252:33621 (34 consecutive failures): Timed out: Failed to ping master at 127.4.171.252:33621: Ping RPC to 127.4.171.252:33621 timed out after 1.000s (SENT)
I20250110 20:58:54.535336 8523 raft_consensus.cc:491] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:54.536022 8523 raft_consensus.cc:513] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:54.538349 8523 leader_election.cc:290] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035), 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745)
W20250110 20:58:54.539706 7436 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9ef4ce3d44cc486fb0ee0ad85127011b (127.4.171.194:34035): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
W20250110 20:58:54.540242 7435 leader_election.cc:343] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 6008006d70c44ad7b5f6667e6b5c2d5f
I20250110 20:58:54.540592 7435 leader_election.cc:304] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c3b63d95ca734d8a9803b96bfd94f307; no voters: 244d9ba396f549f2a5015792b6c5fff8, 9ef4ce3d44cc486fb0ee0ad85127011b
I20250110 20:58:54.541254 8523 raft_consensus.cc:2743] T 6008006d70c44ad7b5f6667e6b5c2d5f P c3b63d95ca734d8a9803b96bfd94f307 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:54.599128 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:54.647158 8526 raft_consensus.cc:491] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250110 20:58:54.647635 8526 raft_consensus.cc:513] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "244d9ba396f549f2a5015792b6c5fff8" member_type: VOTER last_known_addr { host: "127.4.171.193" port: 36745 } } peers { permanent_uuid: "9ef4ce3d44cc486fb0ee0ad85127011b" member_type: VOTER last_known_addr { host: "127.4.171.194" port: 34035 } } peers { permanent_uuid: "c3b63d95ca734d8a9803b96bfd94f307" member_type: VOTER last_known_addr { host: "127.4.171.195" port: 45717 } }
I20250110 20:58:54.649533 8526 leader_election.cc:290] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745), c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717)
W20250110 20:58:54.650749 7287 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 244d9ba396f549f2a5015792b6c5fff8 (127.4.171.193:36745): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
W20250110 20:58:54.651189 7288 leader_election.cc:343] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c3b63d95ca734d8a9803b96bfd94f307 (127.4.171.195:45717): Not found: Tablet not found: 623a0b5cf6964f8294a6153510054ac0
I20250110 20:58:54.651551 7288 leader_election.cc:304] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9ef4ce3d44cc486fb0ee0ad85127011b; no voters: 244d9ba396f549f2a5015792b6c5fff8, c3b63d95ca734d8a9803b96bfd94f307
I20250110 20:58:54.652160 8526 raft_consensus.cc:2743] T 623a0b5cf6964f8294a6153510054ac0 P 9ef4ce3d44cc486fb0ee0ad85127011b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250110 20:58:54.801061 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:54.804756 8531 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:54.805354 8531 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:54.807307 8531 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:54.807897 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:54.808269 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:54.809589 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:54.810232 8531 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:55.705217 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:55.967686 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:56.201387 8535 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:56.201865 8535 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:56.204262 8535 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:56.204810 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:56.205250 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:56.210549 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:56.211437 8535 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:56.680833 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:58:56.762683 8541 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:56.763140 8541 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:56.764927 8541 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:56.765947 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:56.766112 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:56.767241 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:56.767845 8541 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:57.191061 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:57.760371 8547 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:57.760742 8547 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:57.762578 8547 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:57.763319 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:57.763592 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:57.764695 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:57.765168 8547 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:57.834266 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:58.231401 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:58.386806 8551 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:58.387326 8551 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:58.389114 8551 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:58.389698 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:58.389982 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:58.392299 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:58.392863 8551 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:58.765682 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:58:59.402741 6842 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
I20250110 20:58:59.427855 8562 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:58:59.428313 8562 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:58:59.431298 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:58:59.434592 8562 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:58:59.435834 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:58:59.436882 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:58:59.437394 8562 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:58:59.580086 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:59:00.432237 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:59:00.449379 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:59:01.238760 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:59:01.608645 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
I20250110 20:59:02.190939 8575 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:59:02.191301 8575 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:59:02.193068 8575 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:59:02.194185 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:59:02.194120 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:59:02.195667 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:59:02.196765 8575 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20250110 20:59:02.349985 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 72de597ddab241fbbb82fc04e40385c5
W20250110 20:59:02.505115 6839 catalog_manager.cc:5359] Ignoring report from unknown tablet 69f0a5ceae3a489b9b0552c550d3e80a
W20250110 20:59:02.782964 6992 rpcz_store.cc:267] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33774 (request call id 95) took 59999 ms (client timeout 59999 ms). Trace:
W20250110 20:59:02.783367 6992 rpcz_store.cc:269] 0110 20:58:02.783362 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:02.803665 (+ 20303us) service_pool.cc:225] Handling call
0110 20:58:02.821653 (+ 17988us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b54000316d4. stack: 00007f728d56d853 00007f728d56d750 00007f728d0e2584 00000000004fcd24 00000000004fccb9 00007f729446dd50 00007f72944f0f2a 00007f7296195dec 00007f729628f986 00007f7293ffc644 00007f7293ffc5d2 00007f7293ffc501 00007f7293ffc47c 00007f7293ffb732 00007f72922b5dfc 00007f72922b5236
0110 20:58:03.853082 (+1031429us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:59:02.782753 (+58929671us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":687568,"Hive Metastore.run_cpu_time_us":1733,"Hive Metastore.run_wall_time_us":654750,"spinlock_wait_cycles":36725504}
W20250110 20:59:02.785005 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 139) took 57773 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.785247 6992 rpcz_store.cc:269] 0110 20:58:05.012017 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.784644 (+57772627us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.784935 (+ 291us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.785997 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 125) took 56947 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.786247 6992 rpcz_store.cc:269] 0110 20:58:05.839092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.785832 (+56946740us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.785931 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.786799 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 140) took 56764 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.787073 6992 rpcz_store.cc:269] 0110 20:58:06.022772 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.786651 (+56763879us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.786751 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.787369 7565 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.4.171.252:33621)
W20250110 20:59:02.787673 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 126) took 55942 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.787911 6992 rpcz_store.cc:269] 0110 20:58:06.845529 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.787524 (+55941995us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.787629 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.788600 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 141) took 55765 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.788874 6992 rpcz_store.cc:269] 0110 20:58:07.023109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.788449 (+55765340us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.788557 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.789474 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 127) took 54932 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.789775 6992 rpcz_store.cc:269] 0110 20:58:07.856937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.789306 (+54932369us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.789430 (+ 124us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.790437 6988 rpcz_store.cc:267] Call kudu.master.MasterService.AlterTable from 127.0.0.1:33774 (request call id 96) took 59999 ms (client timeout 59999 ms). Trace:
W20250110 20:59:02.790962 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 142) took 54759 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.791256 6992 rpcz_store.cc:269] 0110 20:58:08.032071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.790811 (+54758740us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.790922 (+ 111us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.791924 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 128) took 53915 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.792207 6992 rpcz_store.cc:269] 0110 20:58:08.876973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.791760 (+53914787us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.791871 (+ 111us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.790742 6988 rpcz_store.cc:269] 0110 20:58:02.791133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:58:02.803751 (+ 12618us) service_pool.cc:225] Handling call
0110 20:58:02.823623 (+ 19872us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b54000316d4. stack: 00007f728d56d853 00007f728d56d750 00007f728d0e2584 00000000004fcd24 00000000004fccb9 00007f729446dd50 00007f72944f0f2a 00007f7296195dec 00007f729628f986 00007f7293ffc644 00007f7293ffc5d2 00007f7293ffc501 00007f7293ffc47c 00007f7293ffb732 00007f72922b5dfc 00007f72922b5236
0110 20:58:03.854076 (+1030453us) catalog_manager.cc:2348] Looking up, locking, and authorizing table
0110 20:59:02.790311 (+58936235us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":1341400,"Hive Metastore.run_cpu_time_us":1768,"Hive Metastore.run_wall_time_us":316418,"spinlock_wait_cycles":44160896}
W20250110 20:59:02.792953 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 143) took 53755 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.793221 6992 rpcz_store.cc:269] 0110 20:58:09.038026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.792784 (+53754758us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.792901 (+ 117us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.793614 6988 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 129) took 52913 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.793918 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 144) took 52750 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.793911 6988 rpcz_store.cc:269] 0110 20:58:09.880055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.793454 (+52913399us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.793555 (+ 101us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.794246 6992 rpcz_store.cc:269] 0110 20:58:10.044038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.793769 (+52749731us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.793875 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.794701 6988 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 130) took 51908 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.794894 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 145) took 51745 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.795183 6992 rpcz_store.cc:269] 0110 20:58:11.049810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.794745 (+51744935us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.794841 (+ 96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.795868 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 131) took 50403 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.796154 6992 rpcz_store.cc:269] 0110 20:58:12.392498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.795708 (+50403210us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.795810 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.796808 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 146) took 50241 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.797114 6992 rpcz_store.cc:269] 0110 20:58:12.555751 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.796669 (+50240918us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.796767 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.797803 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 147) took 48959 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.795051 6988 rpcz_store.cc:269] 0110 20:58:10.886072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.794527 (+51908455us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.794655 (+ 128us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.798081 6992 rpcz_store.cc:269] 0110 20:58:13.838169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.797653 (+48959484us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.797751 (+ 98us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.798616 6988 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 132) took 48901 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.798735 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.193:58881 (request call id 148) took 47454 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.798947 6988 rpcz_store.cc:269] 0110 20:58:13.897054 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.798446 (+48901392us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.798553 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.799026 6992 rpcz_store.cc:269] 0110 20:58:15.344267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.798596 (+47454329us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.798693 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.799697 6992 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.171.194:59945 (request call id 133) took 47396 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.799746 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 149) took 45952 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.801431 6992 rpcz_store.cc:269] 0110 20:58:15.402990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.799534 (+47396544us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.799655 (+ 121us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.802181 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 134) took 45896 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.801677 6988 rpcz_store.cc:269] 0110 20:58:16.847737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.799568 (+45951831us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.799676 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.802484 6992 rpcz_store.cc:269] 0110 20:58:16.906211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.802019 (+45895808us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.802127 (+ 108us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.803256 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 150) took 44452 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.803545 6992 rpcz_store.cc:269] 0110 20:58:18.350435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.803064 (+44452629us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.803191 (+ 127us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.804255 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 135) took 44396 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.804670 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 151) took 42952 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.805028 6988 rpcz_store.cc:269] 0110 20:58:19.852292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.804432 (+42952140us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.804614 (+ 182us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.804549 6992 rpcz_store.cc:269] 0110 20:58:18.408396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.804099 (+44395703us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.804212 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.806279 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 136) took 42895 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.806412 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 152) took 41452 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.806564 6992 rpcz_store.cc:269] 0110 20:58:19.910469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.806119 (+42895650us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.806214 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.806797 6988 rpcz_store.cc:269] 0110 20:58:21.354417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.806263 (+41451846us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.806369 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.807515 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 137) took 41395 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.807853 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 153) took 39951 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.807844 6992 rpcz_store.cc:269] 0110 20:58:21.411787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.807375 (+41395588us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.807469 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.808300 6988 rpcz_store.cc:269] 0110 20:58:22.856290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.807540 (+39951250us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.807783 (+ 243us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.809037 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 138) took 39895 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.809386 6992 rpcz_store.cc:269] 0110 20:58:22.913360 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.808850 (+39895490us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.808991 (+ 141us) inbound_call.cc:173] Queueing failure response
Metrics: {}
F20250110 20:59:02.810138 7576 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: failed to wait for Hive Metastore notification log listener to catch up: timed out while waiting for the callback to be called
W20250110 20:59:02.810487 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 139) took 38395 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.810752 6992 rpcz_store.cc:269] 0110 20:58:24.415528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.810343 (+38394815us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.810440 (+ 97us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.811568 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 154) took 38450 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.811758 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 155) took 36952 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.812058 6992 rpcz_store.cc:269] 0110 20:58:25.859534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.811619 (+36952085us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.811714 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.812938 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 140) took 36896 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.813202 6992 rpcz_store.cc:269] 0110 20:58:25.917028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.812797 (+36895769us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.812897 (+ 100us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.811918 6988 rpcz_store.cc:269] 0110 20:58:24.358328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.808903 (+38450575us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.809024 (+ 121us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.814031 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 156) took 35453 ms (client timeout 1000 ms). Trace:
*** Check failure stack trace: ***
*** Aborted at 1736542742 (unix time) try "date -d @1736542742" if you are using GNU date ***
W20250110 20:59:02.814339 6988 rpcz_store.cc:269] 0110 20:58:27.361122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.813875 (+35452753us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.813985 (+ 110us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.814525 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 141) took 35395 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.815104 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 157) took 33952 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.815377 6992 rpcz_store.cc:269] 0110 20:58:27.418703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.814366 (+35395663us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.814482 (+ 116us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.815438 6988 rpcz_store.cc:269] 0110 20:58:28.862536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.814954 (+33952418us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.815058 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.817396 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 142) took 33896 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.817752 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 158) took 32453 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.818150 6992 rpcz_store.cc:269] 0110 20:58:30.363931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.817603 (+32453672us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.817707 (+ 104us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.817984 6988 rpcz_store.cc:269] 0110 20:58:28.921142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.817166 (+33896024us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.817319 (+ 153us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.819325 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 143) took 32396 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.819628 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 159) took 30953 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.819880 6988 rpcz_store.cc:269] 0110 20:58:30.423264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.819069 (+32395805us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.819255 (+ 186us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.820006 6992 rpcz_store.cc:269] 0110 20:58:31.866314 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.819493 (+30953179us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.819585 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.821234 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 144) took 30896 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.821683 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 160) took 29453 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.821969 6992 rpcz_store.cc:269] 0110 20:58:33.367979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.821545 (+29453566us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.821641 (+ 96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.821808 6988 rpcz_store.cc:269] 0110 20:58:31.924882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.820963 (+30896081us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.821119 (+ 156us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.823195 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 145) took 29396 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.824607 6988 rpcz_store.cc:269] 0110 20:58:33.427263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.822926 (+29395663us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.823125 (+ 199us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.825313 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 146) took 27896 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.823489 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 161) took 27954 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.825613 6988 rpcz_store.cc:269] 0110 20:58:34.929000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.825136 (+27896136us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.825269 (+ 133us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.825699 6992 rpcz_store.cc:269] 0110 20:58:34.869289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.823354 (+27954065us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.823440 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
PC: @ 0x0 (unknown)
W20250110 20:59:02.826471 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 162) took 26455 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.826823 6992 rpcz_store.cc:269] 0110 20:58:36.371444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.826328 (+26454884us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.826427 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.827394 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 147) took 26396 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.827553 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 163) took 24955 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.827862 6992 rpcz_store.cc:269] 0110 20:58:37.872437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.827406 (+24954969us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.827511 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.828510 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 148) took 24895 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.828786 6992 rpcz_store.cc:269] 0110 20:58:37.932742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.828373 (+24895631us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.828468 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.829496 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 164) took 23455 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.829806 6992 rpcz_store.cc:269] 0110 20:58:39.373904 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.829310 (+23455406us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.829452 (+ 142us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.827761 6988 rpcz_store.cc:269] 0110 20:58:36.431131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.827235 (+26396104us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.827341 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.830484 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 149) took 23396 ms (client timeout 1000 ms). Trace:
*** SIGABRT (@0x3e8000012af) received by PID 4783 (TID 0x7f58ebaf8700) from PID 4783; stack trace: ***
W20250110 20:59:02.830883 6992 rpcz_store.cc:269] 0110 20:58:39.434587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.830330 (+23395743us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.830443 (+ 113us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.831154 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 165) took 21955 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.831676 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 150) took 21894 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.831552 6988 rpcz_store.cc:269] 0110 20:58:40.875266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.830984 (+21955718us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.831090 (+ 106us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.832013 6992 rpcz_store.cc:269] 0110 20:58:40.936978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.831524 (+21894546us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.831631 (+ 107us) inbound_call.cc:173] Queueing failure response
Metrics: {}
@ 0x2dac70 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
W20250110 20:59:02.832777 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 151) took 20393 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.832733 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 166) took 20455 ms (client timeout 1000 ms). Trace:
@ 0x2dd153 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
W20250110 20:59:02.833063 6992 rpcz_store.cc:269] 0110 20:58:42.439643 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.832644 (+20393001us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.832738 (+ 94us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.833158 6988 rpcz_store.cc:269] 0110 20:58:42.377595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.832552 (+20454957us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.832680 (+ 128us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.833781 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 167) took 18955 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.833963 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 152) took 18891 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.834077 6992 rpcz_store.cc:269] 0110 20:58:43.878971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.833640 (+18954669us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.833736 (+ 96us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.834264 6988 rpcz_store.cc:269] 0110 20:58:43.943079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.833798 (+18890719us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.833917 (+ 119us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.834805 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 168) took 17454 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.834980 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 153) took 17389 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.835075 6992 rpcz_store.cc:269] 0110 20:58:45.380488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.834661 (+17454173us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.834760 (+ 99us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.835271 6988 rpcz_store.cc:269] 0110 20:58:45.445198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.834821 (+17389623us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.834938 (+ 117us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.835723 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 169) took 15952 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.836001 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 154) took 15888 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.836006 6992 rpcz_store.cc:269] 0110 20:58:46.883752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.835581 (+15951829us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.835684 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.836354 6988 rpcz_store.cc:269] 0110 20:58:46.947481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.835847 (+15888366us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.835958 (+ 111us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.836740 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 170) took 14451 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.837028 6992 rpcz_store.cc:269] 0110 20:58:48.384881 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.836594 (+14451713us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.836699 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.837698 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 171) took 12950 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.837824 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 155) took 14388 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.837987 6992 rpcz_store.cc:269] 0110 20:58:49.886931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.837548 (+12950617us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.837650 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.838136 6988 rpcz_store.cc:269] 0110 20:58:48.449245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.837668 (+14388423us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.837770 (+ 102us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250110 20:59:02.838178 6839 catalog_manager.cc:3344] Servicing AlterTable request from {username='slave'} at 127.0.0.1:60462:
table { table_name: "default.Table_59954cb851e943ff9f10bca64b86d8b8" } new_table_name: "default.Table_c3ed0987ec6e41d39ddc47cba082a635" modify_external_catalogs: true
W20250110 20:59:02.838665 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 156) took 12887 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.838842 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 172) took 11449 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.838927 6992 rpcz_store.cc:269] 0110 20:58:49.951122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.838525 (+12887403us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.838626 (+ 101us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.839583 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 157) took 11386 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.839849 6992 rpcz_store.cc:269] 0110 20:58:51.452911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.839456 (+11386545us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.839542 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.840472 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 173) took 9949 ms (client timeout 1000 ms). Trace:
@ 0x7f58ffb4a980 (unknown) at ??:0
W20250110 20:59:02.840772 6992 rpcz_store.cc:269] 0110 20:58:52.891055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.840338 (+9949283us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.840433 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.841398 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 158) took 9886 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.841702 6992 rpcz_store.cc:269] 0110 20:58:52.954951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.841255 (+9886304us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.841350 (+ 95us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.842279 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 174) took 8449 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.842545 6992 rpcz_store.cc:269] 0110 20:58:54.393460 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.842165 (+8448705us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.842244 (+ 79us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.843183 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 159) took 8386 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.843443 6992 rpcz_store.cc:269] 0110 20:58:54.456678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.843055 (+8386377us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.843141 (+ 86us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.844058 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 175) took 6949 ms (client timeout 1000 ms). Trace:
@ 0x7f58fad63fb7 gsignal at ??:0
W20250110 20:59:02.839152 6988 rpcz_store.cc:269] 0110 20:58:51.389031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.838698 (+11449667us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.838801 (+ 103us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.844318 6992 rpcz_store.cc:269] 0110 20:58:55.894827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.843939 (+6949112us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.844018 (+ 79us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.844934 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 160) took 6886 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.845194 6992 rpcz_store.cc:269] 0110 20:58:55.958440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.844814 (+6886374us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.844896 (+ 82us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.845465 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 176) took 5448 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.845850 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 161) took 5385 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.845757 6988 rpcz_store.cc:269] 0110 20:58:57.396659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.845305 (+5448646us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.845397 (+ 92us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.846119 6992 rpcz_store.cc:269] 0110 20:58:57.460376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.845730 (+5385354us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.845810 (+ 80us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.846711 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 177) took 3948 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.846956 6992 rpcz_store.cc:269] 0110 20:58:58.898234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.846584 (+3948350us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.846673 (+ 89us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.847550 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 162) took 3885 ms (client timeout 1000 ms). Trace:
@ 0x7f58fad65921 abort at ??:0
W20250110 20:59:02.847815 6992 rpcz_store.cc:269] 0110 20:58:58.962324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.847433 (+3885109us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.847512 (+ 79us) inbound_call.cc:173] Queueing failure response
Metrics: {}
@ 0x2e0c66 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
W20250110 20:59:02.848392 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 178) took 2448 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.848655 6992 rpcz_store.cc:269] 0110 20:59:00.399872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.848275 (+2448403us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.848353 (+ 78us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.849262 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 163) took 2382 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.849545 6992 rpcz_store.cc:269] 0110 20:59:00.466506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.849144 (+2382638us) service_pool.cc:212] Skipping call since client already timed out
0110 20:59:02.849225 (+ 81us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250110 20:59:02.849714 6988 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.193:58881 (request call id 179) took 948 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.850208 6992 rpcz_store.cc:267] Call kudu.master.MasterService.Ping from 127.4.171.194:59945 (request call id 164) took 882 ms (client timeout 1000 ms). Trace:
W20250110 20:59:02.850503 6992 rpcz_store.cc:269] 0110 20:59:01.967582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.850065 (+882483us) service_pool.cc:225] Handling call
0110 20:59:02.850171 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:59:02.851220 6992 master_service.cc:946] ReplaceTablet: received request to replace tablet 1c82412f7ba946b9b0dcffd8fa883353 from {username='slave'} at 127.0.0.1:33774
I20250110 20:59:02.851459 7398 heartbeater.cc:346] Connected to a master server at 127.4.171.252:33621
W20250110 20:59:02.850040 6988 rpcz_store.cc:269] 0110 20:59:01.901389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0110 20:59:02.849533 (+948144us) service_pool.cc:225] Handling call
0110 20:59:02.849670 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250110 20:59:02.854209 6988 master_service.cc:946] ReplaceTablet: received request to replace tablet fb8d5bf6a9694240b10e1b1c178b528b from {username='slave'} at 127.0.0.1:33774
I20250110 20:59:02.856973 7255 heartbeater.cc:346] Connected to a master server at 127.4.171.252:33621
@ 0x7f58fde1eda4 google::LogMessage::SendToLog() at ??:0
@ 0x7f58fde1f950 google::LogMessage::Flush() at ??:0
@ 0x7f58fde24a8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x35d396 kudu::MasterStressTest::AlterTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38abbe _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3840
@ 0x38aa71 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38a1bb std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2dab30 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f58ffb3f6db start_thread at ??:0
@ 0x7f58fae4671f clone at ??:0
I20250110 20:59:03.099315 8597 raft_consensus.cc:491] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5496a4fb46f84c47a170dfe492f14b51)
I20250110 20:59:03.099787 8597 raft_consensus.cc:513] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5496a4fb46f84c47a170dfe492f14b51" member_type: VOTER last_known_addr { host: "127.4.171.254" port: 36415 } } peers { permanent_uuid: "f9bb61bdf30044a1b482d6a66e29435f" member_type: VOTER last_known_addr { host: "127.4.171.253" port: 40203 } } peers { permanent_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" member_type: VOTER last_known_addr { host: "127.4.171.252" port: 33621 } }
I20250110 20:59:03.102344 8597 leader_election.cc:290] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 5496a4fb46f84c47a170dfe492f14b51 (127.4.171.254:36415), a36f292a1a20461e9aef5aefe8fd0f08 (127.4.171.252:33621)
I20250110 20:59:03.102463 6851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "5496a4fb46f84c47a170dfe492f14b51" is_pre_election: true
I20250110 20:59:03.103253 7004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f9bb61bdf30044a1b482d6a66e29435f" candidate_term: 7 candidate_status { last_received { term: 6 index: 195 } } ignore_live_leader: false dest_uuid: "a36f292a1a20461e9aef5aefe8fd0f08" is_pre_election: true
I20250110 20:59:03.104521 8028 leader_election.cc:304] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9bb61bdf30044a1b482d6a66e29435f; no voters: 5496a4fb46f84c47a170dfe492f14b51, a36f292a1a20461e9aef5aefe8fd0f08
I20250110 20:59:03.105178 8597 raft_consensus.cc:2743] T 00000000000000000000000000000000 P f9bb61bdf30044a1b482d6a66e29435f [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority